Unfortunately its not very good at longer context lengths, which sort of defeats the point of efficient scaling with context. See twitter.com/arankomatsuzaki/status/1639…... Its also not really an RNN. The best way to describe the key time mixing operation is a normalized exponentially weighted moving average (EMA)- no non-linearity. Once viewed this way, its not surprising that it struggles at longer contexts- everything decays, and it has limited space to put things. Of course, it does have some clever tricks, and can choose to remember things for a while by upweighting them, but not forever.

Here's a list of tools for scaling up transformer context that have github repos: * FlashAttention: In my experience, the current best solution for n² attention, but it's very hard to scale up beyond the low tens of thousands of tokens. Memory use is O(n) but compute is O(n²). Code: github.com/HazyResearch/flash-attention * Heinsen Routing: In my experience, the current best solution for n×m attention, i.e., mapping n tokens to m tokens. It's like a souped-up version of attention. I've used it to pull up more than a million tokens as context. Memory use and compute are O(nm). It works, but in my (limited) experience, it doesn't work out-of-the-box as well as FlashAttention for n² attention. Code: github.com/glassroom/heinsen_routing * RWKV: A sort-of-recurrent model which claims to have performance comparable to n² attention in transformers. In my (limited) experience, it doesn't. Others seem to agree: twitter.com/arankomatsuzaki/status/1639…... . Code: github.com/BlinkDL/RWKV-LM * RMT (this method): I'm skeptical that the recurrent connections will work as well as n² attention or n×m routing in practice, but I'm going to give it a try. Code: github.com/booydar/t5-experiments/tree…... In addition, the group that developed FlashAttention is working on state-space models (SSMs) that look promising to me. The idea is to approximate n² attention dynamically using only O(n log n) compute. There's no code available, but here's a blog post about it: hazyresearch.stanford.edu/blog/2023-03…... [CORRECTION: Code is available. See comment by lucidrains below. I'm hopeful this will go to the top of my list.] If anyone here has other suggestions for working with long sequences (hundreds of thousands to millions of tokens), I'd love to learn about them.

twitter.com/arankomatsuzaki/status/1639…...

Alas, it doesn't appear to work well for longer contexts: twitter.com/arankomatsuzaki/status/1639…... Has anyone here experimented with this recently to confirm?