How attention sinks keep language models stable
hanlab.mit.edu
How attention sinks keep language models stable
1–10 of 38 posts
Re: How attention sinks keep language models stable
#2> The first few tokens often carried minimal semantic information—sometimes just a start-of-sequence marker or common words like "the" or "a."
I wonder if it makes sense to use the first word as a title of sorts rather than going straight in grammatically correct sentence when prompting
Re: How attention sinks keep language models stable
#3> Researchers had observed similar patterns in BERT, where "a surprisingly large amount of attention focuses on the delimiter token [SEP] and periods," which they argued was used by the model as a sort of no-op. The same summer at Meta, researchers studying vision transformers found similar behavior, observing that models would repurpose uninformative background patches as computational scratchpads.
This seems to go beyond just transformers. For example, I recall reading a paper a while ago that showed a similar effect in an image to image model with a GAN/U-Net architecture [1].
Re: How attention sinks keep language models stable
#4This is nice and useful because the new GPT-OSS model uses this technique. Kudos to the original authors!
Re: How attention sinks keep language models stable
#5> Barbero et al. have shown that attention sinks serve as "pressure valves" preventing what researchers call "over-mixing"—a pathological state where deep models processing long sequences blur important distinctions between tokens. The presence of a sink draws attention away from other tokens, limiting the spread of information (and noise) and resulting in more stable embeddings.
This sounds like it is working for the wrong reasons. Surely the right behavior is for the right neurons to receive attention rather than the first handful. Jamming everything there is the complementary sin of blurring. I would investigate attention equalization paired with a sparsity prior or something similar to prevent blurring.
Re: How attention sinks keep language models stable
#6> Barbero et al. have shown that attention sinks serve as "pressure valves" preventing what researchers call "over-mixing"—a pathological state where deep models processing long sequences blur important distinctions between tokens. The presence of a sink draws attention away from other tokens, limiting the spread of information (and noise) and resulting in more stable embeddings. This sounds like it is working for th…
The point is that there's not always a right token to attend to. If the information you're looking for is not there, no clever attention scheme will find it. The best you can hope for when that happens is that the value returned in the "not found" case is distinguishable from the "found" case. Having an attention sink serve as a fixed "not found" value is one way to do this.
Re: How attention sinks keep language models stable
#7> Barbero et al. have shown that attention sinks serve as "pressure valves" preventing what researchers call "over-mixing"—a pathological state where deep models processing long sequences blur important distinctions between tokens. The presence of a sink draws attention away from other tokens, limiting the spread of information (and noise) and resulting in more stable embeddings. This sounds like it is working for th…
The point is that there's not always a right token to attend to. If the information you're looking for is not there, no clever attention scheme will find it. The best you can hope for when that happens is that the value returned in the "not found" case is distinguishable from the "found" case. Having an attention sink serve as a fixed "not found" value is one way to do this.
Good point. Does that make them mitigate hallucinations?
Re: How attention sinks keep language models stable
#8Seems like this was a better solution to the same problem https://www.evanmiller.org/attention-is-off-by-one.html
Re: How attention sinks keep language models stable
#9This is nice and useful because the new GPT-OSS model uses this technique. Kudos to the original authors!
And, as always, the FOSS ecosystem moves quickly, llama.cpp already fully support them! https://github.com/ggml-org/llama.cpp/pull/15157
Re: How attention sinks keep language models stable
#10Earlier quoted context omitted.
The point is that there's not always a right token to attend to. If the information you're looking for is not there, no clever attention scheme will find it. The best you can hope for when that happens is that the value returned in the "not found" case is distinguishable from the "found" case. Having an attention sink serve as a fixed "not found" value is one way to do this.
Good point. Does that make them mitigate hallucinations?
In a sense? As the article notes, models trained using standard attention develop attention sinks naturally and removing them makes the model deteriorate completely, so the hallucinations you're thinking of were most likely output by a model that had already mitigated them in this way.