Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Attention Is Off By One

evanmiller.org

Post reply on HN
Print view View original

61–70 of 347 posts

Re: Attention Is Off By One

#61

Post by jrochkind1 »

While not about AI or the algorithm mentioned, on the subject of little errors that you can't convince anyone are errors....

In 2011, I wanted to copy the reddit ranking algorithm in a project of my own, so I went to source code to look at it... the algorithm in the source code I found wasn't doing anything at all sensible with negative-sum voted posts.

I thought I discovered the error, some terms swapped in the simple equation, the sign for positive/negative was misapplied.

I blogged it, and [posted it to reddit](https://www.reddit.com/r/programming/comments/td4tz/reddits_...), only to have MANY people, including reddit employees, tell me I am definitely definitely wrong, and the algorithm was working as intended. And that I was in fact not the first to notice what I thought I noticed, and point it out, and be told by everyone I was wrong.

OK, I didn't really understand what was going on, I couldn't make sense of the algorithm if it wasn't wrong, but so be it. I updated my blog post to say that people smarter than me said there was no error in the reddit algorithm, all I can say is this variation makes more sense to me.

Then, three years later in 2014, a commit was made to the reddit source code with exactly the correction I (and others before me) had suggested all along. The one that everyone piled on to tell me how dare I have the temerity to suggest reddit source code is wrong.

https://github.com/reddit-archive/reddit/commit/50d35de04b92...

¯\_(ツ)_/¯

Open source means there are lots of eyes that can find bugs, but sometimes they can't convince anyone they've found a bug. (And of course, then reddit close-sourced their code in 2017).

I never did end up using the ranking feature in my own project, that I had wanted to copy from reddit. I didn't end adding "vote" features to the app.

Re: Attention Is Off By One

#62

Post by sedael »

fwlr wrote: post #24

The author identifies a real problem and poses a simple solution. It passes all my crank tests (why did no one come up with this before? Because the author is intimately familiar with the softmax function from work outside of ML, and plausibly nobody who’s investigating these issues is remotely as familiar, so despite researchers narrowing the issue down to “something to do with softmax”, they don’t have a deep enoug…

>why did no one come up with this before

So it turns out someone did. Specifically google did. This exact same idea has been in flaxformers since at least November 2021.

https://github.com/google/flaxformer/blame/ee62754ebe5a5eeb1...

Specifically to save people a click it says:

> """Softmax function with an additional virtual logit equal to zero.

  For compatibility with some previously trained models.

  This is equivalent to adding one to the denominator.
  In the context of attention, it allows you to attend to nothing.
And creates the exact same modified softmax as this essay. I suppose only time will tell why it was ignored publicly before, maybe it doesn't do much, maybe it just fell through the cracks, maybe google just didnt push it, who knows

Re: Attention Is Off By One

#63

Post by sbszllr »

lostmsu wrote: post #51
sbszllr wrote: post #23

OP is right in that his change would make the softmax in the attention output zero if it "has nothing to add" (QuietAttention, as he said). Buuut, it's missing the forest for the trees. The goal of the last step of attention (ref., Fig. 2, left in https://arxiv.org/abs/1706.03762 ) is not to add/say anything (as the author is saying) but to compute the relationship between the tokens (QK^T) and V -- in layman terms,…

What's wrong with unscaled matrix multiplication? Softmax has some kind of intuition in the context, but why not layer norm or something else instead (if anything is needed at all)?

The family of sigmoid functions has nice gradient properties with theoretical backing. Good starting read: https://stats.stackexchange.com/questions/162988/why-sigmoid...

Re: Attention Is Off By One

#64

Post by refulgentis »

sebzim4500 wrote: post #3

Don't transformers typically have a token at the beginning of the prompt? This seems equivalent to letting the network attend to this token, and produce a zero value if that's what it wants.

not a token, and not the transformers, but yes, commercial chat models are fine-tuned on text transcripts containing dialogues. (i believe llama-2 was as well)

Re: Attention Is Off By One

#65

Post by LoganDark »

janalsncm wrote: post #16

I follow the argument but the proof of the pudding is in the eating. I don’t know what “battles” the author lost to PyTorch lately but a good test would be to modify one of the smaller models (maybe nanogpt) and swap out all of the softmax calls for his quiet softmax. I didn’t see anything relevant on alternatives to softmax, since TFA is specifically questioning softmax in a multihead attention context. Ultimately,…

> a good test would be to modify one of the smaller models (maybe nanogpt) and swap out all of the softmax calls for his quiet softmax.

You'd have to train the model with the quiet softmax before inferencing with it would work.

Re: Attention Is Off By One

#66

Post by marcyb5st »

Imnimo wrote: post #50

>The problem with using softmax is that it forces each attention head to make an annotation, even if it has no information to add to the output vector. Using softmax to choose among discrete alternatives is great; using it for optional annotation (i.e. as input into addition) is, like, not cool, man. The problem here is exacerbated with multi-head attention, as a specialized head is more likely to want to “pass” than…

But you are wasting some of the model's capacity to learn to ignore some of that information. I think it wouldn't hurt. However, if I followed the reasoning correctly, I think the biggest win is to reduce the range of the weights more than improving performance.

> This is what’s been happening in LLMs – for reasons that are only partially understood, Transformer models contain these outlier weights and are emitting Black Swan mega-activations that are much, much, much larger, like orders of magnitude larger, than their peers ...

meaning that once quantized you can either have a finer quantization since the range of possible values is smaller or you can pick a coarser strategy that saves bits for each weight.

Re: Attention Is Off By One

#67

Post by naillo »

Reading this I'm mostly thankful real brain power and the general smart programming community is seriously taking a close look at all these things. I barely feel the need to try to compete for insight gathering it feels very healtily analyzed from every perspective finally.

Re: Attention Is Off By One

#68

Post by sp332 »

int_19h wrote: post #57
sp332 wrote: post #44

Earlier quoted context omitted.

Chat-tuned ones do, but the base models don't. For example, Llama doesn't, but Alpaca has "### Instruction:", "### Input:", and "### Response:".

Base LLaMA still has dedicated tokens for beginning/end of string. What you're describing is the instruction format, which is separate.

Oh, I had misunderstood something.

Re: Attention Is Off By One

#69

Post by sebzim4500 »

keskival wrote: post #58

I thought everyone knew that softmax (and specifically exp functions in it) are poison. I have always worked around them, for example by using large epsilons (approaching one actually), and using low-order polynomial approximations for the exp functions. I thought everyone does that, because you don't need to work long with these models to get NaNs, and when you check why you see it's because of the exp functions. Th…

It is pretty easy to avoid NaNs when working with softmax, you certainly don't need any epsilons. Just subtract the largest value from everything, and you will have no rounding problems or catastrophic cancellation.

Clearly softmax is not too bad, if it is used extensively in all the most powerful models.

Re: Attention Is Off By One

#70

Post by sebzim4500 »

refulgentis wrote: post #64
sebzim4500 wrote: post #3

Don't transformers typically have a token at the beginning of the prompt? This seems equivalent to letting the network attend to this token, and produce a zero value if that's what it wants.

not a token, and not the transformers, but yes, commercial chat models are fine-tuned on text transcripts containing dialogues. (i believe llama-2 was as well)

Are you sure? I have never seen an LLM that did not have a special token for start of text, I'm certain that llama had one and I don't remember anywhere in the llama-2 paper where they said they removed it.
Post reply on HN