My somewhat facetious take is that LLMs are trying really hard to reinvent RNNs and would do so if we just gave them the tools to do so.
Efficient streaming language models with attention sinks
21–30 of 75 posts
Re: Efficient streaming language models with attention sinks
#22How do any of these sliding window techniques handle instructions that are non expected and only show up at the end? For example imagine feeding a book to the model and the last sentence being the instruction “return the count of the letter m in the previous input”. A human would handle this by first letting out an exasperated sigh but then restarting the reading while counting. An LLM has no ability to loop back and…
The example seems like a weird edge case. I don't even know if current models are capable of this in a short input.
Re: Efficient streaming language models with attention sinks
#23On a first quick pass, this looks so good that I'm wondering if it's too good to be true ! But the work looks to be of decent quality and the technique is remarkably straightforward: The idea is to apply attention over the first token and a sliding context window, ignoring everything in-between, in each layer. By implication, each layer must be gradually shifting relevant information forward in the sequence, enabling…
The end of the sequence could be padded with constant "neutral" values?
Re: Efficient streaming language models with attention sinks
#24We introduce StreamingLLM, an efficient framework that enables LLMs trained with a finite length attention window to generalize to infinite sequence length without any fine-tuning. We show that StreamingLLM can enable Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language modeling with up to 4 million tokens and more.
Sorry, what does "up to 4 million tokens and more" mean? It seems like a contradiction.
Re: Efficient streaming language models with attention sinks
#25We introduce StreamingLLM, an efficient framework that enables LLMs trained with a finite length attention window to generalize to infinite sequence length without any fine-tuning. We show that StreamingLLM can enable Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language modeling with up to 4 million tokens and more.
Sorry, what does "up to 4 million tokens and more" mean? It seems like a contradiction.
Here's a reference describing what a context window for LLMs is:
Re: Efficient streaming language models with attention sinks
#26We introduce StreamingLLM, an efficient framework that enables LLMs trained with a finite length attention window to generalize to infinite sequence length without any fine-tuning. We show that StreamingLLM can enable Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language modeling with up to 4 million tokens and more.
Sorry, what does "up to 4 million tokens and more" mean? It seems like a contradiction.
Not really a contradiction so much as redundant/poorly worded. Should have said, "at least 4 million tokens".
Re: Efficient streaming language models with attention sinks
#27We introduce StreamingLLM, an efficient framework that enables LLMs trained with a finite length attention window to generalize to infinite sequence length without any fine-tuning. We show that StreamingLLM can enable Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language modeling with up to 4 million tokens and more.
[deleted]
Re: Efficient streaming language models with attention sinks
#28My somewhat facetious take is that LLMs are trying really hard to reinvent RNNs and would do so if we just gave them the tools to do so.
Yes, indeedy.
Many things learned over the last three decades with smaller (the current terminology is "extremely tiny"! :) neural networks are being revisited for these large models.
Re: Efficient streaming language models with attention sinks
#29How do any of these sliding window techniques handle instructions that are non expected and only show up at the end? For example imagine feeding a book to the model and the last sentence being the instruction “return the count of the letter m in the previous input”. A human would handle this by first letting out an exasperated sigh but then restarting the reading while counting. An LLM has no ability to loop back and…
[deleted]
Re: Efficient streaming language models with attention sinks
#30How do any of these sliding window techniques handle instructions that are non expected and only show up at the end? For example imagine feeding a book to the model and the last sentence being the instruction “return the count of the letter m in the previous input”. A human would handle this by first letting out an exasperated sigh but then restarting the reading while counting. An LLM has no ability to loop back and…
Does an LLM need to loop back to re-read its input, even in a regular (read non-sliding) context window? Maybe I'm misunderstanding, but doesn't the hidden state solve the "lookup" problem in this case? In the sense that the LLM needs to ingest your entire input anyway before answering, then whether your instruction is at the front or at the end carries little impact besides on attention.
It's my understanding that in regular non-sliding window context models the llm is able to pay attention to any part of the input when generating the output. The attention head is essentially able to jump back and forward to any point in its context window. This is what differentiates the attention mechanism from other models that use token proximity as a proxy for relevance.