I don't understand this well enough to say if it is correct, but I do understand it well enough to say it is important if correct.
I don't know half of you half as well as I should like; and I like less than half of you half as well as you deserve
Attention Is Off By One
11–20 of 347 posts
Re: Attention Is Off By One
#12I don't really understand the subject matter enough, so I apologize in advance for the meta-comment... The author mentions that he would maybe have written this as a scientific paper: > I tried writing a serious-looking research paper about the bug and my proposed fix, but I lost a series of pitched battles against Pytorch and biblatex, so I figured I’d just write a blog post instead. (History is written by the winne…
This is why folks like gwern have their own research published this way, i.e. his analysis of GPT-3: https://gwern.net/gpt-3
We call him an "independent AI researcher" because his google scholar is "bland" compared to many academics who play the academia game - https://scholar.google.com/citations?user=yk1QMowAAAAJ&hl=en
Re: Attention Is Off By One
#13TL;DR: The author proposes that instead of using the Softmax function in each head,
Softmax(x_i) = exp(x_i) / sum(exp(x_i)),
Softmax_1(x_i) = exp(x_i) / (1 + sum(exp(x_i))),
Giving each transformer head the ability to ignore all tokens surely can't hurt, but it remains to be seen if it will actually improve transformer performance.
Re: Attention Is Off By One
#14This makes sense. One tweak for the press: I think it would be an improvement to call it OptionalAttention rather than QuietAttention since the goal is to permit an attention head to opt-out.
You might attract more, ahem, attention if it was immediately apparent from the name only what this attention head does that the current one does not. There's also that small matter of distinguishing the internal vs output softmax functions.
Re: Attention Is Off By One
#15I don't really understand the subject matter enough, so I apologize in advance for the meta-comment... The author mentions that he would maybe have written this as a scientific paper: > I tried writing a serious-looking research paper about the bug and my proposed fix, but I lost a series of pitched battles against Pytorch and biblatex, so I figured I’d just write a blog post instead. (History is written by the winne…
I think maybe its because he didn't have experimental results that show that it worked. Not a knock against the author, there are just so many things that seem like good ideas that don't end up working well in practice, a paper like this without results is hard to value.
Yes, definitely. If he tried to have it published, the lack of experimental results would definitely be a glaring error.
But this is still scientific communication. It's really nice that it's legible!
> Even though softmax1 is facially quite boring, I’m 99.44% sure that it will resolve the outlier feedback loop that’s making quantization the subject of cascades of research. If you want to run some experiments and prove me right, DM me on Twitter and we’ll get a paper going.
I'm guessing that in the stodgy world of science, a communication like this might happen over lunch at a conference, limited to a small clique of researchers who are zealously guarding their next paper. Who could blame them, publish or perish!
But someone will probably test this theory out (after my read, it will probably happen in llama.cpp with preliminary results on GPT-2 by next week) and achieve results, and it will happen quickly and legibly to the outside world, because this was published openly and without all of the pretension that formal science (tm) has. If it works, it works. Stuff like this is the soul of the internet. Sharing knowledge and making it legible for all.
Re: Attention Is Off By One
#16I follow the argument but the proof of the pudding is in the eating. I don’t know what “battles” the author lost to PyTorch lately but a good test would be to modify one of the smaller models (maybe nanogpt) and swap out all of the softmax calls for his quiet softmax.
I didn’t see anything relevant on alternatives to softmax, since TFA is specifically questioning softmax in a multihead attention context.
Ultimately, neural networks are arbitrary function approximators. It doesn’t necessarily have to be “right” internally to fit the data. But if this new softmax allows transformers to learn more, that’s great.
Re: Attention Is Off By One
#17I don't really understand the subject matter enough, so I apologize in advance for the meta-comment... The author mentions that he would maybe have written this as a scientific paper: > I tried writing a serious-looking research paper about the bug and my proposed fix, but I lost a series of pitched battles against Pytorch and biblatex, so I figured I’d just write a blog post instead. (History is written by the winne…
I think maybe its because he didn't have experimental results that show that it worked. Not a knock against the author, there are just so many things that seem like good ideas that don't end up working well in practice, a paper like this without results is hard to value.
Then again, if you don't have access to giant compute clusters you can't test this, so it's either a blog post or nothing. I believe the outlier problem that this solves only appears for very large models.
Re: Attention Is Off By One
#18The author says to add a unity vector to the context, i presume of each layer, to not mess with gradient calculations. But most modern DL frameworks compute the gradient for you, (i know this is true for JAX and Pytorch). Is it maybe that hand coded gradient for a well-known enough dl architecture like transformer is faster than letting the framework autodiff it?
Otherwise, i fear some of the 'magic' of transformer networks is that this amplification effect allows it to encode/memorize some results verbatim. And we often are seeing a heavily tuned internet regurgitator. So similar to the rise of RNNs with attention, which supposedly allowed them to focus on some things and ignore others but really often was just overfitting stuff, yielded more interesting results with the overfitting than without.
Re: Attention Is Off By One
#19I ran an experiment like this and in my setting it didn't help. Not saying there may not have been a bug or something, but I think attending over the current position sort of solves this problem. IE when it should not speak it just emits the current pos value. edit to add details in case anyone is interested I didn't add one to the softmax denom. I added a learned parameter (the attention sink) that would be appended…
When you say it didn't help, can you clarify what you're measuring? In the context of this post, I think both the performance your task, and the number of outlier weights (and their magnitude) are important.
Re: Attention Is Off By One
#20If you hesitate to read it, let me say that the post denounces “kurtotic barbarities.” If that expression alone doesn’t convince you to read it, you might not be in the intended audience.