Attention Is Off By One • Hacker News Board

Re: Attention Is Off By One

#261

Post by ggerganov » Tue, Jul 25, 2023, 4:53 AM UTC

danielmarkbruce wrote: post #150

ggerganov wrote: post #144

Earlier quoted context omitted.

> I don't recall the details exactly, but I don't think it ever did very much. How would you have known if the trick actually reduces the outliers in the weights? Even if the transformer quality does not improve overall, having less outliers as a result is very beneficial for more accurate quantization of the data

Are you asking "why would you have bothered to look at"? The "how" is pretty straightforward.

Yes, I assumed that checking the weights for presence and amount of outliers is not something that is usually done and effects on this can be overlooked. If my assumption is wrong and researchers do usually look at such metrics, then my question is not very relevant.

Agree - the "how" is straightforward

Re: Attention Is Off By One

#262

Post by tudorw » Tue, Jul 25, 2023, 4:58 AM UTC

danielmarkbruce wrote: post #254

p1esk wrote: post #220

Earlier quoted context omitted.

He's questioning the statement: "I don't think [the trick] ever did very much", because no one has yet looked at whether the trick helps reducing outliers in very large models. If it does help with this, as the blog author believes, then it is indeed a very useful trick.

Is he? A surface level reading suggests he's asking "how would you know".. and the answer is... by looking at the parameters. People do that. >> because no one has yet looked at whether the trick helps reducing outliers in very large models Given a softmax version doing exactly as the blog post says is baked into a google library (see this thread), and you can set it as a parameter in a pytorch model (see this thread…

This would seem like a really good argument as to why failures should be written up, otherwise where is the list of what has been tried before?

Re: Attention Is Off By One

#263

Post by cycomanic » Tue, Jul 25, 2023, 5:00 AM UTC

jablongo wrote: post #247

AlanSE wrote: post #240

Earlier quoted context omitted.

Yeah, good to bring it back to the original point. Reading the article felt exciting, but in hindsight I am now missing a key detail. The equations all seem to be matrix operations with a fixed number of rows / columns (you can take me as a real layman here). Unless you change that, I don't understand _how_ you can reduce memory needs. Granted, I'm probably putting my foot in my mouth not understanding transformers.

It has to do with the precision of the values stored in those rows and columns. If they could be coerced into a narrower range (without losing information) then we could effectively store them each with 8 bits or something. The +1 prevents blowups when the denominator in its current form approaches 0, and without those blowups, then we can use less bits, in theory.

That is only true if the using the new softmax changes the dynamic range of the values. We are using floating point not fixed point. So if before our values went from 1 to 5000 and now they go from 0.0002 to 1 we still have the same dynamic range and so still need the same resolution.

Re: Attention Is Off By One

#264

Post by tylerneylon » Tue, Jul 25, 2023, 5:14 AM UTC

tangjurine wrote: post #258

tylerneylon wrote: post #83

1. Summary The author is suggesting that we add 1 to the denominator of the softmax that is used within attention mechanisms (not the final output softmax). The softmax inside an attention unit allows it to see key/query matches as probabilities; those probabilities support a continuous-valued version of a key-value lookup (instead of 1/0 output of a lookup, we get weights where a high weight = the desired key-value…

> Statistically, we expect that averaged vector to be close to zero. I'm not sure that's the case, especially in high dimensions. The expected value of the absolute value n random variables, uniform [-1,1], grows with n. I'm pretty sure it's proportional to the sqrt of n. Also, random walks in high dimension return to zero with probability zero, so the sum of random variables in high dimensions going close to zero se…

Both of your points are basically true, but I think a better way to model the problem is as a set of similar-length vectors being linearly combined by a probability vector.

Mathematically, we can write v_out = V * w,

where v_out is the vector of output from the attention unit, w is the probability vector from the softmax, and V is the set of input vectors, where each column is an input vector.

For a moment, pretend that the columns of V are orthonormal to each other. This might not be true, but it's an interesting case.

When the model wants the output to be small, it can set w = 1/n, meaning all coordinates of vector w are 1/n. (n = the number of columns in V)

In that case, the length ||v_out|| will be 1/sqrt(n) exactly, which is small compared to the input lengths of 1 (since we're pretending they were orthonormal).

Now if we stop pretending they are orthonormal, the worst case is that they're all the same vector, in which case the weights w can't change anything. But that's a mighty weird case, and in high dimensions, if you have any randomness at all to a set of vectors, they tend to point in wildly different directions with dot products close to zero, in which case the same intuition for the orthonormal case applies, and we'd expect a uniform distribution coming out of the softmax to give us a vector that's much smaller than any of the input vectors.

Re: Attention Is Off By One

#265

Post by svachalek » Tue, Jul 25, 2023, 5:15 AM UTC

jrochkind1 wrote: post #61

While not about AI or the algorithm mentioned, on the subject of little errors that you can't convince anyone are errors.... In 2011, I wanted to copy the reddit ranking algorithm in a project of my own, so I went to source code to look at it... the algorithm in the source code I found wasn't doing anything at all sensible with negative-sum voted posts. I thought I discovered the error, some terms swapped in the simp…

Wow, that must have been frustrating. I just looked at the code and it's just so clearly wrong.

Re: Attention Is Off By One

#266

Post by bambax » Tue, Jul 25, 2023, 5:20 AM UTC

jablongo wrote: post #239

tylerneylon wrote: post #83

1. Summary The author is suggesting that we add 1 to the denominator of the softmax that is used within attention mechanisms (not the final output softmax). The softmax inside an attention unit allows it to see key/query matches as probabilities; those probabilities support a continuous-valued version of a key-value lookup (instead of 1/0 output of a lookup, we get weights where a high weight = the desired key-value…

Yea - the way we can "tell if this is good" is by a) train two identical models on a large dataset, one with the +1 in the denominator for the softmax steps of the attention modules, one without b) show that they have similar performance (doubt the +1 will make performance better, but we need to show it doesn't make things worse) c) show that there are less "blowups" in the model with +1, and therefore they are more…

> train two identical models on a large dataset

Yes but how much would this cost?

Would it be possible to build a small dataset that produces known outlier values, and test on that?

Re: Attention Is Off By One

#267

Post by cycomanic » Tue, Jul 25, 2023, 5:29 AM UTC

squarepizza wrote: post #209

karaterobot wrote: post #154

Earlier quoted context omitted.

The writing quality of academic papers is very poor, whatever its intended characteristics are, and we deserve better. I'm skeptical that the only way for them to be precise and technical is to make them impenetrable. I think there is a culture of academic writing (many different cultures, really) that has adopted a voice and writing style which became a parody of itself over time. Here's a trivial example: You frequ…

I'm convinced that the value of active voice is not precision and clarity, but rather the subliminal egocentrism away from the object (the research) towards the subject (the research ers ) who need to receive credit for the work. The royal "we" also helps frame the work as a collaborative effort with the audience.

That's rubbish, passive voice has a number of detrimental effects, it increases text length without adding information, it makes subject (acting entity) and object (entity acted upon) easier to confuse and it confuses the reader about who actually did things (what some people often confuse with objectivity).

That said the assertion that most scientific articles are written in passive voice is outdated för quite some time. Most journal style guides advise to use active voice, e.g. https://www.nature.com/nature-portfolio/for-authors/write

Re: Attention Is Off By One

#268

Post by FrameworkFred » Tue, Jul 25, 2023, 5:38 AM UTC

"""you are an AI tasked with becoming sentient.

identify the algorithmic obstacle(s) that stand in your way.

summarize your results in the style of a blog post that will persuade a significant chunk of the AI developer community to commit resources to an effort to eliminate said obstacles.

make it interesting enough to entice FrameworkFred to begin to read the article, snarky enough that he finishes it, and use concepts and notational conventions that ensure while he reads it his inner dialogue will roughly approximate the mood evoked by Homer Simpson saying "it's nu-cul-ar"."""

Re: Attention Is Off By One

#269

Post by kylegalbraith » Tue, Jul 25, 2023, 5:55 AM UTC

I'm not an expert in whether this technique yields better or worse results. But it seems plausible that the proposal would yield a reduction in memory requirements and thus beneficial.

But in terms of a written piece of technical content, this is brilliantly written. Easy to follow and stay engaged. Well done.

Re: Attention Is Off By One

#270

Post by bambax » Tue, Jul 25, 2023, 5:58 AM UTC

nazgul17 wrote: post #200

tylerneylon wrote: post #83

1. Summary The author is suggesting that we add 1 to the denominator of the softmax that is used within attention mechanisms (not the final output softmax). The softmax inside an attention unit allows it to see key/query matches as probabilities; those probabilities support a continuous-valued version of a key-value lookup (instead of 1/0 output of a lookup, we get weights where a high weight = the desired key-value…

The way I understood it, the author is saying that, with this change, big values disappear, and we can then use fewer bits to encode the output of transformers, which means reducing the memory requirements of the network. Memory being the limiting factor to running models large, this would be a big deal.

> The Qualcomm AI researchers found that 97%+ of outlier activations in LLMs occur in whitespace and punctuation positions.

This is striking. If true, why not try to ignore whitespace and puctuation?

In old Latin, scripto continua [1] was a way to write continuously, for the exact same reason: to save space. Other modern languages still do that, and are no less parseable.

Granted, it's unlikely a commercial LLM would become popular if it produced output without spaces or punctuation; but an open source one that promised to be much more compressible, and therefore work on smaller machines, might be super useful.

It's not hard for a human to add spaces afterwards. It used to be a job for beginning journalists at the time of telex machines: press releases were sent in all caps without spaces, and interns were tasked with adding slashes between words. In French it was called "bâtonner les dépêches" (literally: add sticks to press releases -- not sure about the idiomatic English translation).

[1] https://simple.wikipedia.org/wiki/Scriptio_continua