Seems like just an extension of Short-conv in the linear attention methods where we instead carry the EMA of the entire sequence instead of nearest 4 tokens.
It would make sense in my opinion to do the evals based on the reason for needing Short-conv in the architectures that use it in the first place, where it was added to support exact retrieval/needle-in-haystack when we compress the attention state.
It's odd to make this arch extension and not test beyond a 2k context window, or with the needle-in-haystack benchmark.
In my head this should fail because the EMA cannot carry this granular information
Seems like just an extension of Short-conv in the linear attention methods where we instead carry the EMA of the entire sequence instead of nearest 4 tokens.
It would make sense in my opinion to do the evals based on the reason for needing Short-conv in the architectures that use it in the first place, where it was added to support exact retrieval/needle-in-haystack when we compress the attention state.
It's odd to make this arch extension and not test beyond a 2k context window, or with the needle-in-haystack benchmark.
In my head this should fail because the EMA cannot carry this granular information