Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Transformer to 1M tokens and beyond with RMT

arxiv.org

Post reply on HN
Print view View original

111–120 of 147 posts

Re: Scaling Transformer to 1M tokens and beyond with RMT

#111

Post by pclmulqdq »

avereveard wrote: post #91
nl wrote: post #26

For those who don't completely get the impact of this: We already know Large Language Models (LLMs) can learn at runtime (ie, separately to the training process.) This is called "In Context Learning". See [1], [2] for more details. (BTW, when anyone says "LLMs are stochastic parrots" you know they are ignorant of this) In context learning is wonderful because it means you can "train" a LLM at run time by filling the…

It's a hard problem that token space only partially solve, because with filling the token space to the brim with documents will dilute the instructions in the prompt. One may argue that we'd need a separate channel for the instructions, but it would make harder to do anything but zero shot question and answer. I think it's likely that systems with metadata for tagging sources, tasks separately from the human and ai i…

It sounds like that's a good use for encoder-decoder architecture. GPT-3 and LLaMA are decoder-only architectures.

Training that encoder sounds hard, though.

Re: Scaling Transformer to 1M tokens and beyond with RMT

#112

Post by cs702 »

Here's a list of tools for scaling up transformer context that have github repos:

* FlashAttention: In my experience, the current best solution for n² attention, but it's very hard to scale up beyond the low tens of thousands of tokens. Memory use is O(n) but compute is O(n²). Code: https://github.com/HazyResearch/flash-attention

* Heinsen Routing: In my experience, the current best solution for n×m attention, i.e., mapping n tokens to m tokens. It's like a souped-up version of attention. I've used it to pull up more than a million tokens as context. Memory use and compute are O(nm). It works, but in my (limited) experience, it doesn't work out-of-the-box as well as FlashAttention for n² attention. Code: https://github.com/glassroom/heinsen_routing

* RWKV: A sort-of-recurrent model which claims to have performance comparable to n² attention in transformers. In my (limited) experience, it doesn't. Others seem to agree: https://twitter.com/arankomatsuzaki/status/16390003799784038... . Code: https://github.com/BlinkDL/RWKV-LM

* RMT (this method): I'm skeptical that the recurrent connections will work as well as n² attention or n×m routing in practice, but I'm going to give it a try. Code: https://github.com/booydar/t5-experiments/tree/scaling-repor...

In addition, the group that developed FlashAttention is working on state-space models (SSMs) that look promising to me. The idea is to approximate n² attention dynamically using only O(n log n) compute. There's no code available, but here's a blog post about it: https://hazyresearch.stanford.edu/blog/2023-03-27-long-learn... [CORRECTION: Code is available. See comment by lucidrains below. I'm hopeful this will go to the top of my list.]

If anyone here has other suggestions for working with long sequences (hundreds of thousands to millions of tokens), I'd love to learn about them.

Re: Scaling Transformer to 1M tokens and beyond with RMT

#113

Post by tough »

nzach wrote: post #108
DebtDeflation wrote: post #92

Earlier quoted context omitted.

>if you wanted to say load all of a companies documents and ask questions This is literally the use case that every large enterprise wants. Why? The potential cost savings are enormous. A significant percentage of white collar jobs primarily involve performing repetitive tasks and answering frequently asked questions. If you could automate even 20% of this........

This is already happening right now. Companies will find a way to put all their knowledge inside a LLM. The problem is that I don't think they are ready for what this will cause. Imagine that Amazon managed to do that and now every employee has access to it. What prevents anyone asking "inconvenient" questions? Think something like: - Does the Echo dot records audio even when the trigger word isn't used? Show me the…

It's quite easy if you go the vector embedding route, and don't output any info the authenticated user doesn't have access too

The info lives in a database, not the LLM' model weights, you just use the LLM to understand natural langauge questions, find relevant documents (the user has access to) and providing relevant bits from them to the context to allow responding with them (That's how all systems who show sources work afaik, braid, bing, and mdx docs based stuff)

For outsiders? That's just public FAQ's

You're the CEO? You can ask anything.

Re: Scaling Transformer to 1M tokens and beyond with RMT

#114

Post by lucidrains »

cs702 wrote: post #112

Here's a list of tools for scaling up transformer context that have github repos: * FlashAttention: In my experience, the current best solution for n² attention, but it's very hard to scale up beyond the low tens of thousands of tokens. Memory use is O(n) but compute is O(n²). Code: https://github.com/HazyResearch/flash-attention * Heinsen Routing: In my experience, the current best solution for n×m attention, i.e.,…

the code is here https://github.com/hazyresearch/safari you should try it and let us know your verdict.

Re: Scaling Transformer to 1M tokens and beyond with RMT

#115

Post by drakmo »

avereveard wrote: post #91
nl wrote: post #26

For those who don't completely get the impact of this: We already know Large Language Models (LLMs) can learn at runtime (ie, separately to the training process.) This is called "In Context Learning". See [1], [2] for more details. (BTW, when anyone says "LLMs are stochastic parrots" you know they are ignorant of this) In context learning is wonderful because it means you can "train" a LLM at run time by filling the…

It's a hard problem that token space only partially solve, because with filling the token space to the brim with documents will dilute the instructions in the prompt. One may argue that we'd need a separate channel for the instructions, but it would make harder to do anything but zero shot question and answer. I think it's likely that systems with metadata for tagging sources, tasks separately from the human and ai i…

The LLM should categorize the prompts and answers and consider this for evaluation of further responses. It could give instructions a "more local weight" for the context but e.g. knowledge be it generated or provided a "more global weight". The impact of instructions or other local context parts would get lesser and lesser. This could be a function of time or length of conversation. Probably methods like this one are already tested with "reflection" behaviours of LLMs.

Re: Scaling Transformer to 1M tokens and beyond with RMT

#116

Post by [deleted] »

albertzeyer wrote: post #34

This looks like a potential NeurIPS submission. But it will probably be rejected. The quality bar for NeurIPS is quite high. Some reasons: The experiments are very weak: There are just a few figures, basically figure 1 and figure 5, which show some results. There are no tables with numbers. But more importantly: There are no comparisons (in terms of experiments/numbers) to similar models, like: - Block-Recurrent Tran…

[deleted]

Re: Scaling Transformer to 1M tokens and beyond with RMT

#117

Post by cs702 »

lucidrains wrote: post #114
cs702 wrote: post #112

Here's a list of tools for scaling up transformer context that have github repos: * FlashAttention: In my experience, the current best solution for n² attention, but it's very hard to scale up beyond the low tens of thousands of tokens. Memory use is O(n) but compute is O(n²). Code: https://github.com/HazyResearch/flash-attention * Heinsen Routing: In my experience, the current best solution for n×m attention, i.e.,…

the code is here https://github.com/hazyresearch/safari you should try it and let us know your verdict.

Thank you. Somehow I missed that. I'm still making my way through the related papers, trying to get a superficial but accurate/intuitive understanding of these models. Embarrassingly, the work hasn't quite 'clicked' for me yet. Looking forward to tinkering with the code!

Re: Scaling Transformer to 1M tokens and beyond with RMT

#118

Post by DebtDeflation »

tough wrote: post #113
nzach wrote: post #108

Earlier quoted context omitted.

This is already happening right now. Companies will find a way to put all their knowledge inside a LLM. The problem is that I don't think they are ready for what this will cause. Imagine that Amazon managed to do that and now every employee has access to it. What prevents anyone asking "inconvenient" questions? Think something like: - Does the Echo dot records audio even when the trigger word isn't used? Show me the…

It's quite easy if you go the vector embedding route, and don't output any info the authenticated user doesn't have access too The info lives in a database, not the LLM' model weights, you just use the LLM to understand natural langauge questions, find relevant documents (the user has access to) and providing relevant bits from them to the context to allow responding with them (That's how all systems who show sources…

The idea is to get away from Snippet + URL and instead just get a natural language answer to your question that uses knowledge from that internal corpus.

Re: Scaling Transformer to 1M tokens and beyond with RMT

#119

Post by danicgross »

cs702 wrote: post #112

Here's a list of tools for scaling up transformer context that have github repos: * FlashAttention: In my experience, the current best solution for n² attention, but it's very hard to scale up beyond the low tens of thousands of tokens. Memory use is O(n) but compute is O(n²). Code: https://github.com/HazyResearch/flash-attention * Heinsen Routing: In my experience, the current best solution for n×m attention, i.e.,…

cs702, fantastic comment. I am sorta poking around this area too. I'd be curious what benchmark you're using to evaluate performance amongst these repos? If you're up for it, shoot me an email -- my email is in my profile.

Re: Scaling Transformer to 1M tokens and beyond with RMT

#120

Post by cs702 »

danicgross wrote: post #119
cs702 wrote: post #112

Here's a list of tools for scaling up transformer context that have github repos: * FlashAttention: In my experience, the current best solution for n² attention, but it's very hard to scale up beyond the low tens of thousands of tokens. Memory use is O(n) but compute is O(n²). Code: https://github.com/HazyResearch/flash-attention * Heinsen Routing: In my experience, the current best solution for n×m attention, i.e.,…

cs702, fantastic comment. I am sorta poking around this area too. I'd be curious what benchmark you're using to evaluate performance amongst these repos? If you're up for it, shoot me an email -- my email is in my profile.

Thank you!

Working on proprietary stuff. Not allowed to share details.

But I'll ask about connecting online :-)

Post reply on HN