Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Attention Is Off By One

evanmiller.org

Post reply on HN
Print view View original

191–200 of 347 posts

Re: Attention Is Off By One

#191

Post by renewiltord »

nl wrote: post #186
renewiltord wrote: post #146

Earlier quoted context omitted.

Do you know of handy testing steps? I suppose I could ask ChatGPT, but if someone has a validated "here, this is how you do it" I have a 3090 that I can do it on, but I'm not keen to debug anything here.

Testing steps (based on thinking about this for 30 seconds - so probably can be improved): Train a Transformer based model with and without the modified Softmax (Suggestions: GPT-2 or nanoGPT) Measure performance - I'd probably start with Perplexity and see if there is any difference (we'd expect little difference). Quantize both models with different quantization strategies. Measure the perplexity of the quantized m…

I was thinking about a different problem as I was typing that and got some mental memory alias bug. I wanted to know a set of steps to take to train a model. My apologies.

In any case, that was an lmgtfy-level question. Here's what I found: https://til.simonwillison.net/llms/training-nanogpt-on-my-bl...

I shall try that soon.

Re: Attention Is Off By One

#192

Post by llwu »

tsurba wrote: post #161

In the text they say you need to cram all information needed to predict the next token into a single 6KB word embedding, but isn’t that wrong? Rather, isn’t the autoregressively predicted single next token a combination (based on attention) of all 6KB word tokens in the attention window. So the size of memory where all information for next token prediction needs to be ”crammed into” is more like window_size*6KB, righ…

This is probably what the author meant to say but elided. I can see why it looks off though.

Re: Attention Is Off By One

#193

Post by thomasahle »

mlsu wrote: post #8

I don't really understand the subject matter enough, so I apologize in advance for the meta-comment... The author mentions that he would maybe have written this as a scientific paper: > I tried writing a serious-looking research paper about the bug and my proposed fix, but I lost a series of pitched battles against Pytorch and biblatex, so I figured I’d just write a blog post instead. (History is written by the winne…

> it does not have an "ego" the way scientific papers tend to have.

What do you call it when somebody takes the time to write about "a big discovery" they've made, but don't take the time to check if somebody else already did it? It's not like it's in some forgotten paper nobody has seen. It's in Pytorch itself.

Also this: "I’m 99.44% sure that it will resolve the outlier feedback loop that’s making quantization the subject of cascades of research."

Re: Attention Is Off By One

#195

Post by llwu »

sebzim4500 wrote: post #3

Don't transformers typically have a token at the beginning of the prompt? This seems equivalent to letting the network attend to this token, and produce a zero value if that's what it wants.

This is a cool idea - it could be tested it by seeing whether W_V tends to map to 0, and whether tends to get attended to.

There are interesting things to said whether or not this turns out to be the case.

Re: Attention Is Off By One

#196

Post by sweezyjeezy »

tylerneylon wrote: post #83

1. Summary The author is suggesting that we add 1 to the denominator of the softmax that is used within attention mechanisms (not the final output softmax). The softmax inside an attention unit allows it to see key/query matches as probabilities; those probabilities support a continuous-valued version of a key-value lookup (instead of 1/0 output of a lookup, we get weights where a high weight = the desired key-value…

> First, if an attention node has low confidence, it can already assign similar scores pre-softmax. Then we get what looks like a uniform distribution as output.

Disagree here, I think neural nets are quite bad at implicitly learning low entropy transforms, similar to how they struggle to model the identity function, necessitating residual connections. In both cases the change doesn't increase expressivity, but it does bake these needle-in-a-haystack transformations into the model that may be hard to access with gradient descent.

Can't speak to how useful it is though.

Re: Attention Is Off By One

#197

Post by radq »

nl wrote: post #174
quickthrower2 wrote: post #173

I wonder why the author didn't try it on some metal? What is the minimum cost to get information that would say "this is better" or at least "this has a good chance of being better, spending a million training a bigger model is worth it"? Could you rent a bank for 8 A100's for a day and try it out on a smaller model and prove something. Not cheap, but doesn't need VC money either. Probably about $400 on LambdaLabs to…

You could test this on a toy Transformer-based model trainable on a consumer GPU.

Do outlier features emerge in sub-100M parameter models? I haven't seen any research discuss it below the 124M scale (bert-base). At that scale training a model takes ~4 days on an 8xA100 node.

Re: Attention Is Off By One

#199

Post by janalsncm »

ersiees wrote: post #116

This trick “they found” is part of the standard torch implementation of multi head attention, namely it is called, add_zero_attention. They add a zero to the logits, resulting in a one in the denominator as e^0=1 https://pytorch.org/docs/stable/generated/torch.nn.Multihead...

Nice catch! Hopefully OP will see this.

Re: Attention Is Off By One

#200

Post by nazgul17 »

tylerneylon wrote: post #83

1. Summary The author is suggesting that we add 1 to the denominator of the softmax that is used within attention mechanisms (not the final output softmax). The softmax inside an attention unit allows it to see key/query matches as probabilities; those probabilities support a continuous-valued version of a key-value lookup (instead of 1/0 output of a lookup, we get weights where a high weight = the desired key-value…

The way I understood it, the author is saying that, with this change, big values disappear, and we can then use fewer bits to encode the output of transformers, which means reducing the memory requirements of the network. Memory being the limiting factor to running models large, this would be a big deal.
Post reply on HN