Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Attention Is Off By One

evanmiller.org

Post reply on HN
Print view View original

281–290 of 347 posts

Re: Attention Is Off By One

#281

Post by klabb3 »

madrox wrote: post #158
jrochkind1 wrote: post #61

While not about AI or the algorithm mentioned, on the subject of little errors that you can't convince anyone are errors.... In 2011, I wanted to copy the reddit ranking algorithm in a project of my own, so I went to source code to look at it... the algorithm in the source code I found wasn't doing anything at all sensible with negative-sum voted posts. I thought I discovered the error, some terms swapped in the simp…

When I was an intern at Yahoo working on OAuth back in 2008 (2007? It was long ago and I'm old) I had the pleasure of implementing an internal tool for generating OAuth 1.0 URLs, which meant encoding a lot of things in query parameters. My tool did not generate URLs which were compatible with Yahoo's implementation (certain parameters effectively should be encoded twice, which my tool did). The implementing engineer…

> It literally took bringing in Eran Hammer-Lahav to weigh in on the topic to say I was correct, at which point the engineer agreed that of course that was correct. I got zero acknowledgment or apology for the days of ad hominem attacks against me.

If it weren’t for the torturous gaslighting, this is borderline hilarious. Appeal-to-authority types have a way of submitting so effortlessly when a grander poobah comes around. Spine made of jelly.

Re: Attention Is Off By One

#282

Post by uoaei »

AlanSE wrote: post #240
nazgul17 wrote: post #200

Earlier quoted context omitted.

The way I understood it, the author is saying that, with this change, big values disappear, and we can then use fewer bits to encode the output of transformers, which means reducing the memory requirements of the network. Memory being the limiting factor to running models large, this would be a big deal.

Yeah, good to bring it back to the original point. Reading the article felt exciting, but in hindsight I am now missing a key detail. The equations all seem to be matrix operations with a fixed number of rows / columns (you can take me as a real layman here). Unless you change that, I don't understand _how_ you can reduce memory needs. Granted, I'm probably putting my foot in my mouth not understanding transformers.

The activations (outputs) of one layer must be encoded in the same way as the weights of that layer as well as the weights of the next layer or the computation fails (unless you manage to write clever kernels for doing math at different levels of precision simultaneously, but even then you're introducing even more lossiness than just using a binary representation for those values).

Example: multiplying a bunch of float16s together gives you a float16. That is passed on to the next layer of float16s. Why should forcing the output of the first step to be float8 confer any advantage here? The only way I can see this argument working is if you make all the layers float8 too, and the reason you can do that is that the output of the first step can be faithfully represented as float8 because it doesn't ever blow up. If that's what the author is saying, it wasn't very clear.

Re: Attention Is Off By One

#283

Post by uoaei »

sweezyjeezy wrote: post #196
tylerneylon wrote: post #83

1. Summary The author is suggesting that we add 1 to the denominator of the softmax that is used within attention mechanisms (not the final output softmax). The softmax inside an attention unit allows it to see key/query matches as probabilities; those probabilities support a continuous-valued version of a key-value lookup (instead of 1/0 output of a lookup, we get weights where a high weight = the desired key-value…

> First, if an attention node has low confidence, it can already assign similar scores pre-softmax. Then we get what looks like a uniform distribution as output. Disagree here, I think neural nets are quite bad at implicitly learning low entropy transforms, similar to how they struggle to model the identity function, necessitating residual connections. In both cases the change doesn't increase expressivity, but it do…

Surely you mean high-entropy, ie, uniform? We are talking about extremely low-entropy predictions as being the problem here.

Re: Attention Is Off By One

#284

Post by uoaei »

thaumasiotes wrote: post #219
tylerneylon wrote: post #83

1. Summary The author is suggesting that we add 1 to the denominator of the softmax that is used within attention mechanisms (not the final output softmax). The softmax inside an attention unit allows it to see key/query matches as probabilities; those probabilities support a continuous-valued version of a key-value lookup (instead of 1/0 output of a lookup, we get weights where a high weight = the desired key-value…

I actually prefer the conceptual model the author suggests: > Originally I wanted to call this function ghostmax, as you can think of there being an extra zero-valued entry in x (as exp(0)=1), as well as a zero vector in the V matrix that attenuates the result. Don't think of this as weighting the options so that some of the time none of them is chosen. ("Weights that add up to less than 1.") Instead, think of this a…

This seems like a good way to look at it. Another way to put it is, there is a certain "origin" or "default" confidence which is pinned to some fixed value pre-softmax, ie, all outputs are necessarily compared to that fixed value (pretending zero is another input to the softmax) rather than merely each other.

Re: Attention Is Off By One

#285

Post by uoaei »

PartiallyTyped wrote: post #162
mrfox321 wrote: post #125

Earlier quoted context omitted.

You are misreading things. Regardless of numerical stability tricks (e.g. exp(x_i-max(x))), you are still simply normalizing the logits such that the probabilities sum to 1. The blog adds an additional hidden logit (equal to 0) to allow for softmax(x) = 0 when x -> -inf.

How can `x -> -inf` occur in the first place when nearly everything is within [-2,2] and doing a dot product plus before that there's normalization too?

The use of the "nearly" in your comment is exactly occluding the issue as presented.

Enough weights don't fall under that "nearly" that we require more bits per weight to cover those edge cases. If we were able to delete the "nearly" we would need fewer bits (smaller models).

Re: Attention Is Off By One

#286

Post by klabb3 »

Q6T46nT668w6i3m wrote: post #169

This method was frequently used prior to the ubiquity of dummy tokens. XLNet was the paper that introduced me to this idea. I believe it’s been in PyTorch since 2019/2020. I would not be surprised if someone finds an earlier reference. I’m surprised by the pompousness in the OP. Especially about something that most people who do transformer research understand. I’m also surprised that so many in the replies are takin…

> I’m surprised by the pompousness in the OP.

He’s writing in a colloquial and self-deprecating and humorous tone. I can’t speak to the merits, but I can follow the reasoning perfectly fine. It’d be hard to find something further from pompous.

> saving yourself some embarrassment

Implying of course that being wrong, or not the first one to discover this, is embarrassing. And that’s not pompous?

Re: Attention Is Off By One

#288

Post by uoaei »

orasis wrote: post #52

“the seemingly innocent exponentiator that no one thought capable of such kurtotic barbarities.” This writing brought a happy tear to my eye.

It's a funny turn of phrase and also either extremely sarcastic or extremely wrong. Everyone knows about exponentials blowing up. exp(40) or so blows up a float32.

Re: Attention Is Off By One

#289

Post by klabb3 »

doliveira wrote: post #30
mlsu wrote: post #8

I don't really understand the subject matter enough, so I apologize in advance for the meta-comment... The author mentions that he would maybe have written this as a scientific paper: > I tried writing a serious-looking research paper about the bug and my proposed fix, but I lost a series of pitched battles against Pytorch and biblatex, so I figured I’d just write a blog post instead. (History is written by the winne…

Nah, scientific papers are supposed to be precise and technical. This reads like those quite frequent suggestions here of switching all equations in papers to plain English or code: it honestly comes from a place of ignorance, and I say that as basically a layman myself. What should be encouraged is for academics to blog about their research as well. It would even help when recruiting and onboarding new members. Righ…

> Nah, scientific papers are supposed to be precise and technical.

> What should be encouraged is for academics to blog about their research as well.

Why so binary? A blog would be hard to find, why not have both in the paper?

My view is similar to that of code vs docs: code should be as small, and as precise as possible, whereas docs are best when they’re explaining to humans how things fit together, high level. Also easier to maintain.

Hyper technical natural language mixed in with math is almost the worst of both worlds: low density of the actual formulas, with an incomprehensible wall of text surrounding it. And clearly this is an issue also for phd domain experts.

Not saying academic writing could be super simple but I also see no reason that the status quo is optimized more for comprehension than say social posturing.

Re: Attention Is Off By One

#290

Post by PartiallyTyped »

uoaei wrote: post #285
PartiallyTyped wrote: post #162

Earlier quoted context omitted.

How can `x -> -inf` occur in the first place when nearly everything is within [-2,2] and doing a dot product plus before that there's normalization too?

The use of the "nearly" in your comment is exactly occluding the issue as presented. Enough weights don't fall under that "nearly" that we require more bits per weight to cover those edge cases. If we were able to delete the "nearly" we would need fewer bits (smaller models).

So the concern is not that x->-inf due to values but it happens due to numerical issues arising out of lower precision?
Post reply on HN