This reminds me of the normalization bug in StyleGAN. It had this obvious visual artifact of a 'blob' which would appear in otherwise photorealistic images, which was puzzling because it was so obvious how did the Discriminator not squash it? It turned out to be a flaw in the normalization of the AdaIn style layers, IIRC, where the Generator was pumping up numbers and doing weird things to force through information.
Attention Is Off By One
91–100 of 347 posts
Re: Attention Is Off By One
#92I might be missing something obvious, but I am not sure why everyone in the comments think it's a big deal. I've seen this trick in practice multiple times. For example, see this snippet from an old Google repo: https://github.com/google/flaxformer/blob/ee62754ebe5a5eeb11...
If popular models are still making this mistake then it still seems noteworthy and making a blog post or paper to increase awareness definitely seems worthwhile. Also multiple independent discovery of good ideas is quite common.
Re: Attention Is Off By One
#93Earlier quoted context omitted.
When you say it didn't help, can you clarify what you're measuring? In the context of this post, I think both the performance your task, and the number of outlier weights (and their magnitude) are important.
I was just looking at doing this in pretraining, so I was looking at pretraining losses. The difference was within the range of usual noise so I didn't keep trying.
The question concerns outliers ... how did the change manage them?
Re: Attention Is Off By One
#94Can i please ask why lim{x->-inf} softmax(x) = 1/k ?
Re: Attention Is Off By One
#95I ran an experiment like this and in my setting it didn't help. Not saying there may not have been a bug or something, but I think attending over the current position sort of solves this problem. IE when it should not speak it just emits the current pos value. edit to add details in case anyone is interested I didn't add one to the softmax denom. I added a learned parameter (the attention sink) that would be appended…
He's advertising it as fixing the spiking outliers. Did your variant have those outliers beforehand?
Re: Attention Is Off By One
#96This is right below the "Have Attention Spans Been Declining? – Yes, 65%" post, lol brilliant. In general, human decreasing, AI increasing- attention.
"In this post, I prove that attention spans have actually declined by 64%, contrary to widely-publicized reports of 65%..."
Re: Attention Is Off By One
#97I don't really understand the subject matter enough, so I apologize in advance for the meta-comment... The author mentions that he would maybe have written this as a scientific paper: > I tried writing a serious-looking research paper about the bug and my proposed fix, but I lost a series of pitched battles against Pytorch and biblatex, so I figured I’d just write a blog post instead. (History is written by the winne…
Counterargument: this blogpost is worthless. You get all the way to the end and then find out he hasn't actually tried it, not even on a toy model. It's just a neat idea he thinks will work.
I wouldn’t quite say its value is zero. It’s worth something, but a lot less than if it had been shown to work empirically.
Explainers and their folksy, imprecise tone are good for things we already know are true. I’m skeptical on things which are unproven.
Re: Attention Is Off By One
#98TL;DR: The author proposes that instead of using the Softmax function in each head, Softmax(x_i) = exp(x_i) / sum(exp(x_i)), we should use instead what the author calls the Softmax_1 function, Softmax_1(x_i) = exp(x_i) / (1 + sum(exp(x_i))), which would make it possible for each transformer head's attention probabilities to be zero, i.e., attend to nothing, by computing x_i's with values well below zero. Giving each…
I also saw the author distinguished internal versus output softmax. I think he'd apply his modification only to internal softmax and let the external force an output.
Yes, it makes sense to apply this only to the Softmax we use to compute attention. It makes no sense to apply it to the output Softmax, which must compute a probability distribution over the vocabulary.
Re: Attention Is Off By One
#99Interesting read. As others have said, it will be much more convincing with some experimental numbers.
I'm confused what his goal is though:
I could imagine some theoretical reason to add a 1 there, but he starts by saying this can lead to smaller, more compactable models. Is he talking about the size the compressed weights? or pruning to a smaller model? or resistant more quantization?
Parts of the essay seemed to throw me off track, because I'm not sure if they are relevant at all to the proposal (eg the of the initial embedding and how many bits it would take the store the vocab size, etc).
Re: Attention Is Off By One
#100I ran an experiment like this and in my setting it didn't help. Not saying there may not have been a bug or something, but I think attending over the current position sort of solves this problem. IE when it should not speak it just emits the current pos value. edit to add details in case anyone is interested I didn't add one to the softmax denom. I added a learned parameter (the attention sink) that would be appended…
[deleted]