Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How attention sinks keep language models stable

hanlab.mit.edu

Post reply on HN
Print view View original

21–30 of 38 posts

Re: How attention sinks keep language models stable

#21

Post by yorwba »

CamperBob2 wrote: post #19
xg15 wrote: post #13

Earlier quoted context omitted.

Some people start their prompts with "Hello" or "Please" or something similar, out of some habitual sense of politeness, I think. It would be hilarious if those prompts really work better because the model can use those words as attention sinks.

One point that Karpathy has made in some of his videos is that using additional tokens in the prompt can facilitate computation. If you ask a transformer to do some basic math, it will be more likely to get the right answer (or at least a better approximation) with a more verbose prompt. To me, this backs up the use of more conversational language ("Please," etc.) when prompting. However, that seems to be contradicte…

> It's hard to tell where the prompt stops and the CoT response starts, in fact.

That's because you're looking at the final output that includes neither the prompt nor the intermediate chain of thought.

Re: How attention sinks keep language models stable

#22

Post by CamperBob2 »

xg15 wrote: post #20
CamperBob2 wrote: post #19

Earlier quoted context omitted.

One point that Karpathy has made in some of his videos is that using additional tokens in the prompt can facilitate computation. If you ask a transformer to do some basic math, it will be more likely to get the right answer (or at least a better approximation) with a more verbose prompt. To me, this backs up the use of more conversational language ("Please," etc.) when prompting. However, that seems to be contradicte…

I wonder if the model could also just make its own sink tokens if the prompt doesn't have any. E.g. if the model first emits some "fluff" like "The answer to this question is:" before starting with the actual answer, it could use those tokens as attention sinks. Same with "thinking tokens" that don't directly contribute to the answer or invisible formatting tokens, etc.

True, along with "You're absolutely right! What an insightful observation. You're going places, bro," yadda yadda yadda.

It would be amusing if all that gratuitous sycophancy actually helped with inference accuracy. It would also be worth treating that as a bug to be fixed, of course.

Re: How attention sinks keep language models stable

#23

Post by CamperBob2 »

yorwba wrote: post #21
CamperBob2 wrote: post #19

Earlier quoted context omitted.

One point that Karpathy has made in some of his videos is that using additional tokens in the prompt can facilitate computation. If you ask a transformer to do some basic math, it will be more likely to get the right answer (or at least a better approximation) with a more verbose prompt. To me, this backs up the use of more conversational language ("Please," etc.) when prompting. However, that seems to be contradicte…

> It's hard to tell where the prompt stops and the CoT response starts, in fact. That's because you're looking at the final output that includes neither the prompt nor the intermediate chain of thought.

Good point -- I can see that, but it all ends up in the same context, anyway. Point being, the model seems to prefer to conserve tokens.

That said, now I'm wondering if all those dashes it spews out are more than just window dressing.

Re: How attention sinks keep language models stable

#25

Post by SpaceManNabs »

Calavar wrote: post #3

> Researchers had observed similar patterns in BERT, where "a surprisingly large amount of attention focuses on the delimiter token [SEP] and periods," which they argued was used by the model as a sort of no-op. The same summer at Meta, researchers studying vision transformers found similar behavior, observing that models would repurpose uninformative background patches as computational scratchpads. This seems to go…

I miss GANs. I understand that they are much harder to train than transformers for the same performance even with high data regime and high parameter regime, but there was such good optimization research and tricks that came out of them.

The work on the capacity of discriminators was super cool.

Re: How attention sinks keep language models stable

#26

Post by microtonal »

canjobear wrote: post #8

Seems like this was a better solution to the same problem https://www.evanmiller.org/attention-is-off-by-one.html

The attention sink as used in gpt-oss is similar to your link. But rather than adding one to the denominator, they add a trainable 'logit' (a different logit for each head).

Re: How attention sinks keep language models stable

#27

Post by godelski »

SpaceManNabs wrote: post #25
Calavar wrote: post #3

> Researchers had observed similar patterns in BERT, where "a surprisingly large amount of attention focuses on the delimiter token [SEP] and periods," which they argued was used by the model as a sort of no-op. The same summer at Meta, researchers studying vision transformers found similar behavior, observing that models would repurpose uninformative background patches as computational scratchpads. This seems to go…

I miss GANs. I understand that they are much harder to train than transformers for the same performance even with high data regime and high parameter regime, but there was such good optimization research and tricks that came out of them. The work on the capacity of discriminators was super cool.

  > much harder to train than transformers
There's plenty of GANs that use transformers. PWC seems to be redirecting to GitHub currently but IIRC about half of top scores on FFHQ256 were GANs with transformers in them. I know that the number 2 was, I saw it at CVPR. It was a lot smaller and had higher throughput than the diffusion models it was outperforming.

Though the main reason diffusion took over was for the ability to encode more diversity. I still think there's a place for GANs and we overcorrected by putting too much focus on diffusion, but there are a lot of fundamental advantages to diffusion. Though they aren't strictly better, there's no global optima for solution spaces this large. I think the ML community (maybe CS in general) has a tendency to take an all or nothing approach. I don't think this is a really good strategy...

Re: How attention sinks keep language models stable

#30

Post by blackbear_ »

xg15 wrote: post #20
CamperBob2 wrote: post #19

Earlier quoted context omitted.

One point that Karpathy has made in some of his videos is that using additional tokens in the prompt can facilitate computation. If you ask a transformer to do some basic math, it will be more likely to get the right answer (or at least a better approximation) with a more verbose prompt. To me, this backs up the use of more conversational language ("Please," etc.) when prompting. However, that seems to be contradicte…

I wonder if the model could also just make its own sink tokens if the prompt doesn't have any. E.g. if the model first emits some "fluff" like "The answer to this question is:" before starting with the actual answer, it could use those tokens as attention sinks. Same with "thinking tokens" that don't directly contribute to the answer or invisible formatting tokens, etc.

Good thought, that indeed works: https://arxiv.org/abs/2310.02226
Post reply on HN