Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Efficient streaming language models with attention sinks

github.com

Post reply on HN
Print view View original

31–40 of 75 posts

Re: Efficient streaming language models with attention sinks

#31

Post by cs702 »

Nevermark wrote: post #23
cs702 wrote: post #8

On a first quick pass, this looks so good that I'm wondering if it's too good to be true ! But the work looks to be of decent quality and the technique is remarkably straightforward: The idea is to apply attention over the first token and a sliding context window, ignoring everything in-between, in each layer. By implication, each layer must be gradually shifting relevant information forward in the sequence, enabling…

The end of the sequence could be padded with constant "neutral" values?

Wouldn't work. Imagine a sequence with 100 tokens, fed to a model with 10 layers, each with a sliding attention window spanning 5 tokens. The top layer's final sliding window can only see 5 trailing tokens, each of which can only see 5 trailing tokens in the previous layer, and so on, for a total of 50 trailing tokens (plus the initial token) of maximum trailing context in the top layer.

It's an inherent limitation of this approach.

Re: Efficient streaming language models with attention sinks

#32

Post by refulgentis »

alex_duf wrote: post #22
doctoboggan wrote: post #12

How do any of these sliding window techniques handle instructions that are non expected and only show up at the end? For example imagine feeding a book to the model and the last sentence being the instruction “return the count of the letter m in the previous input”. A human would handle this by first letting out an exasperated sigh but then restarting the reading while counting. An LLM has no ability to loop back and…

The example seems like a weird edge case. I don't even know if current models are capable of this in a short input.

I agree, even just tokenization screws you here, I'm 95% sure. I.e. the raw input isn't letters but one of 100K integers that represent some set of letters.

That being said, probably a naive take, since we're seeing them do so much. & I bet we could get it to count correctly with at least some short input, and given infinite runs, probably trivial. (I.e. for N characters, split into N inputs, for each one "say true if it is an M, false otherwise,)

Re: Efficient streaming language models with attention sinks

#33

Post by doctoboggan »

alex_duf wrote: post #22
doctoboggan wrote: post #12

How do any of these sliding window techniques handle instructions that are non expected and only show up at the end? For example imagine feeding a book to the model and the last sentence being the instruction “return the count of the letter m in the previous input”. A human would handle this by first letting out an exasperated sigh but then restarting the reading while counting. An LLM has no ability to loop back and…

The example seems like a weird edge case. I don't even know if current models are capable of this in a short input.

Ignore the specific example of counting characters, I was just quickly coming up with a situation where the instruction is at the end of the input. Here is a better example:

Input the full text of a novel, then ask for a minor detail (eg color of a car that is briefly mentioned in the middle of the book). Again a human can do this by flipping back to the relevant section but LLMs have no mechanism for this when using a sliding window attention scheme.

If the full input can fit in the context window then any LLM today would be able to extract the color of the car.

Re: Efficient streaming language models with attention sinks

#34

Post by obblekk »

iandanforth wrote: post #21

My somewhat facetious take is that LLMs are trying really hard to reinvent RNNs and would do so if we just gave them the tools to do so.

RNNs are the correct solution, but infeasibly expensive to run.

A different way to think about it is Transformer models are trying to predict which part of the RNN network is "worth" keeping given a resource constraint.

Transformers use a simple heuristic today (and this result makes the heuristic better). Just like many NP complete problems, there might be approximations that are not perfectly correct but still useful. Transformers prove that is the case for neural networks.

Re: Efficient streaming language models with attention sinks

#36

Post by anon291 »

iandanforth wrote: post #21

My somewhat facetious take is that LLMs are trying really hard to reinvent RNNs and would do so if we just gave them the tools to do so.

I think many people believe you. The main advantage of transformers over RNNs is training parallelization. RNNs are hard because training suffers from vanishing gradients and also because it's hard to get full utilization (needs large batches to get good utilization).

The existence of models like RWKV indicates that there is potentially a future in training like a transformer but inferring like an RNN.

Re: Efficient streaming language models with attention sinks

#39

Post by doctoboggan »

refulgentis wrote: post #32
alex_duf wrote: post #22

Earlier quoted context omitted.

The example seems like a weird edge case. I don't even know if current models are capable of this in a short input.

I agree, even just tokenization screws you here, I'm 95% sure. I.e. the raw input isn't letters but one of 100K integers that represent some set of letters. That being said, probably a naive take, since we're seeing them do so much. & I bet we could get it to count correctly with at least some short input, and given infinite runs, probably trivial. (I.e. for N characters, split into N inputs, for each one "say true i…

I understand that, which is why I said "Ignore LLM issues with character counting for this example". It was a quick example, please see my other comment with a better example.

Re: Efficient streaming language models with attention sinks

#40

Post by Nevermark »

cs702 wrote: post #31
Nevermark wrote: post #23

Earlier quoted context omitted.

The end of the sequence could be padded with constant "neutral" values?

Wouldn't work. Imagine a sequence with 100 tokens, fed to a model with 10 layers, each with a sliding attention window spanning 5 tokens. The top layer's final sliding window can only see 5 trailing tokens, each of which can only see 5 trailing tokens in the previous layer, and so on, for a total of 50 trailing tokens (plus the initial token) of maximum trailing context in the top layer. It's an inherent limitation o…

How about neutral value padding at the other end?

I am having trouble visualizing this.

Post reply on HN