Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Attention Is Off By One

evanmiller.org

Post reply on HN
Print view View original

301–310 of 347 posts

Re: Attention Is Off By One

#301

Post by yldedly »

Imnimo wrote: post #80
marcyb5st wrote: post #66

Earlier quoted context omitted.

But you are wasting some of the model's capacity to learn to ignore some of that information. I think it wouldn't hurt. However, if I followed the reasoning correctly, I think the biggest win is to reduce the range of the weights more than improving performance. > This is what’s been happening in LLMs – for reasons that are only partially understood, Transformer models contain these outlier weights and are emitting B…

Right, I get the goal of removing the outlier activations, but I just don't understand why outlier activations are a consequence of the model trying to "pass". The story from the linked paper earlier in the post ( https://arxiv.org/pdf/2306.12929.pdf ) is that the model is doing the following: -Learn a near-zero representation for some otherwise low-importance token, like delimiters or whitespace. -When a head wants…

The projection and MLP layers don't compare all embedding pairs like attention does, so they can't distinguish between contexts where delimiters are low- vs high-importance. The projection layer mixes the multi-heads in the same way always, and the same MLP is applied to every input.

Re: Attention Is Off By One

#302

Post by fstokesman »

ersiees wrote: post #116

This trick “they found” is part of the standard torch implementation of multi head attention, namely it is called, add_zero_attention. They add a zero to the logits, resulting in a one in the denominator as e^0=1 https://pytorch.org/docs/stable/generated/torch.nn.Multihead...

https://en.wikipedia.org/wiki/Multiple_discovery

Re: Attention Is Off By One

#303

Post by squarepizza »

cycomanic wrote: post #267
squarepizza wrote: post #209

Earlier quoted context omitted.

I'm convinced that the value of active voice is not precision and clarity, but rather the subliminal egocentrism away from the object (the research) towards the subject (the research ers ) who need to receive credit for the work. The royal "we" also helps frame the work as a collaborative effort with the audience.

That's rubbish, passive voice has a number of detrimental effects, it increases text length without adding information, it makes subject (acting entity) and object (entity acted upon) easier to confuse and it confuses the reader about who actually did things (what some people often confuse with objectivity). That said the assertion that most scientific articles are written in passive voice is outdated för quite some…

> it confuses the reader about who actually did things

When scientific papers have a clear list of authors and delineated section headings, this point is moot. And in such papers, again, repetitive strings of sentences that begin with the same "we..." emphasizes the producers of the work over the work itself.

Re: Attention Is Off By One

#304

Post by TeMPOraL »

bambax wrote: post #270
nazgul17 wrote: post #200

Earlier quoted context omitted.

The way I understood it, the author is saying that, with this change, big values disappear, and we can then use fewer bits to encode the output of transformers, which means reducing the memory requirements of the network. Memory being the limiting factor to running models large, this would be a big deal.

> The Qualcomm AI researchers found that 97%+ of outlier activations in LLMs occur in whitespace and punctuation positions. This is striking. If true, why not try to ignore whitespace and puctuation? In old Latin, scripto continua [1] was a way to write continuously, for the exact same reason : to save space. Other modern languages still do that, and are no less parseable. Granted, it's unlikely a commercial LLM woul…

> This is striking. If true, why not try to ignore whitespace and puctuation?

It is initially, but thinking about it some more, there's a lot of information packed in whitespace and punctuation choice.

Scripto continua may have worked because the few readers who lived back then expected it to encode some form of legal or religious prose, but even then they could learn things from the overall shape of the document. LLMs are working in a much richer domain of document types, but the only thing they can "see" is a stream of tokens. There's no spatial or geometric data attached there. So whitespace and punctuation are the only thing an LLM has to make inferences about otherwise textually identical inputs. Such as:

  (see: other)  -- vs -- {see: other}
One being likely a text fragment, the other likely a piece of code.

Or how spacing may imply Markdown or YAML being used. Or how it may imply a list. Or a poem. Or a song. Or specific writing style, such as "lol im a casual who not care bout comms" vs. "I am a distinguished professor, about to retire. Elites like us put two spaces after full stop."

Re: Attention Is Off By One

#305

Post by dxuh »

bambax wrote: post #270
nazgul17 wrote: post #200

Earlier quoted context omitted.

The way I understood it, the author is saying that, with this change, big values disappear, and we can then use fewer bits to encode the output of transformers, which means reducing the memory requirements of the network. Memory being the limiting factor to running models large, this would be a big deal.

> The Qualcomm AI researchers found that 97%+ of outlier activations in LLMs occur in whitespace and punctuation positions. This is striking. If true, why not try to ignore whitespace and puctuation? In old Latin, scripto continua [1] was a way to write continuously, for the exact same reason : to save space. Other modern languages still do that, and are no less parseable. Granted, it's unlikely a commercial LLM woul…

LLMs are also pretty good at programming and I would expect this to nuke programming ability completely, wouldn't it?

Re: Attention Is Off By One

#306

Post by sweezyjeezy »

uoaei wrote: post #283
sweezyjeezy wrote: post #196

Earlier quoted context omitted.

> First, if an attention node has low confidence, it can already assign similar scores pre-softmax. Then we get what looks like a uniform distribution as output. Disagree here, I think neural nets are quite bad at implicitly learning low entropy transforms, similar to how they struggle to model the identity function, necessitating residual connections. In both cases the change doesn't increase expressivity, but it do…

Surely you mean high-entropy, ie, uniform? We are talking about extremely low-entropy predictions as being the problem here.

yep - always get that the wrong way round haha

Re: Attention Is Off By One

#307

Post by TimPC »

zamalek wrote: post #260
AlanSE wrote: post #240

Earlier quoted context omitted.

Yeah, good to bring it back to the original point. Reading the article felt exciting, but in hindsight I am now missing a key detail. The equations all seem to be matrix operations with a fixed number of rows / columns (you can take me as a real layman here). Unless you change that, I don't understand _how_ you can reduce memory needs. Granted, I'm probably putting my foot in my mouth not understanding transformers.

More ELI5 than the other comments. Considering the softmax network: During quantization we find that values in the network vary from 0->5000, but 95% of values are This is where exotic encodings come into play. We might try to use a logarithmic scheme, for example. This would result in higher value densities at lower values - but we would probably still waste bits and it would require more APU cycles. Now switch to t…

No one quantizes blindly without accounting for data. If 95% of your values are in 0-100 you’ll probably do something like have 20 values for 0-100 and the remaining 12 for 101-5000. You don’t have to apply a uniform distribution and shouldn’t when your data is that concentrated.

Re: Attention Is Off By One

#308

Post by thfuran »

GTP wrote: post #296
colordrops wrote: post #274

Earlier quoted context omitted.

Seems you could make a pipeline where a much simpler model adds spaces and punctuation to output from the main model.

For the spaces and for some (maybe most?) languages you don't even need a NN to add spaces: as words made of two or more words aren't that common, and when those occur you probably want to use the composite one, it boils down to start from the beginning of the text and look in a dictionary what's the longest string that is a valid word. The only language that I know of that uses a lot of composite words (I mean words…

I think you're significantly underestimating how many words could be retokenized into multiple words even before considering how concatenation affects things. For example: Concatenate is a word, but so are con, catenate, cat, and enate. Yes, no two of those are likely to be used in sequence, but I don't think that's a very reliable rule overall—"a" and "an" are both common words and negative prefixes.

Re: Attention Is Off By One

#309

Post by Zetice »

cycomanic wrote: post #221

The I know it's on-vogue on HN to complain about academia, but the blog post is not making a good argument. The post could have probably gotten the point across in less than 1/4 of the overall length (probably even less than 1/8th), instead the author wrapped the the post into lots of informalisms and a thinly veiled complained about academic publishing. The result of this is reflected in the discussion here, nobody…

You sneer at “getting on the front page of HN” but when you rephrase it to, “discuss something you’ve observed informally” your dismissal loses its oomph.

The point may be to entertain as well as inform. Many humans enjoy the unfocused discussion around the main point, and perhaps the author prefers it to the clinical and formal tone an academic paper tends to take.

Re: Attention Is Off By One

#310

Post by AbrahamParangi »

tylerneylon wrote: post #83

1. Summary The author is suggesting that we add 1 to the denominator of the softmax that is used within attention mechanisms (not the final output softmax). The softmax inside an attention unit allows it to see key/query matches as probabilities; those probabilities support a continuous-valued version of a key-value lookup (instead of 1/0 output of a lookup, we get weights where a high weight = the desired key-value…

One caveat is that the average of many normally distributed vectors in many dimensions is normally distributed with 0 mean but is not typically close to 0. In fact the average norm is quite large. Try it yourself and see!
Post reply on HN