Efficient streaming language models with attention sinks
1–10 of 75 posts
Re: Efficient streaming language models with attention sinks
#2We introduce StreamingLLM, an efficient framework that enables LLMs trained with a finite length attention window to generalize to infinite sequence length without any fine-tuning. We show that StreamingLLM can enable Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language modeling with up to 4 million tokens and more.
Re: Efficient streaming language models with attention sinks
#3Okay, what's the downside this time?
Re: Efficient streaming language models with attention sinks
#4So I can let llama2 summarize books now or are there any non-obvious caveats to this approach?
Re: Efficient streaming language models with attention sinks
#5Okay, what's the downside this time?
Allegedly not “efficiency or performance”, though I’m skeptical. Will dig into this later and update my comment (if I remember).
Re: Efficient streaming language models with attention sinks
#6This is a big claim, curious to see what the caveats are.
Re: Efficient streaming language models with attention sinks
#7So can it now understand and write complete applications?
Re: Efficient streaming language models with attention sinks
#8On a first quick pass, this looks so good that I'm wondering if it's too good to be true!
But the work looks to be of decent quality and the technique is remarkably straightforward:
The idea is to apply attention over the first token and a sliding context window, ignoring everything in-between, in each layer.
By implication, each layer must be gradually shifting relevant information forward in the sequence, enabling the top layer's ending sliding attention window to see it.
The only caveat I can think of is that the sliding windows won't be able to shift all important information forward when the span of all sliding windows isn't sufficient to span the entire sequence -- for example, when model depth × window length < sequence length, if all windows have the same length.
Re: Efficient streaming language models with attention sinks
#9I feel like information theory prevents full information retention for unlimited context lengths and finite compute, but I don't know if we are at information theory limits to invoke this argument. Or rather, I don't know how to make a good analysis of (bits of context information) per (bits of model parameters).
Re: Efficient streaming language models with attention sinks
#10Having only read the abstract, I'm probably way off the mark here, but my first thought was: LLM + LSTM.