Please, put empirical numbers with proposals like these. Transformers have had a billion "improvements" suggested to them through the years.
Attention Is Off By One
21–30 of 347 posts
Re: Attention Is Off By One
#22I ran an experiment like this and in my setting it didn't help. Not saying there may not have been a bug or something, but I think attending over the current position sort of solves this problem. IE when it should not speak it just emits the current pos value. edit to add details in case anyone is interested I didn't add one to the softmax denom. I added a learned parameter (the attention sink) that would be appended…
When you say it didn't help, can you clarify what you're measuring? In the context of this post, I think both the performance your task, and the number of outlier weights (and their magnitude) are important.
I was just looking at doing this in pretraining, so I was looking at pretraining losses. The difference was within the range of usual noise so I didn't keep trying.
Re: Attention Is Off By One
#23OP is right in that his change would make the softmax in the attention output zero if it "has nothing to add" (QuietAttention, as he said).
Buuut, it's missing the forest for the trees. The goal of the last step of attention (ref., Fig. 2, left in https://arxiv.org/abs/1706.03762) is not to add/say anything (as the author is saying) but to compute the relationship between the tokens (QK^T) and V -- in layman terms, simplifying, which tokens are related to each other. The softmax is there because it gives a representation that is nicer to work with, it gives probabilities, instead of unscaled matrix multiplication.
TLDR; author isn't wrong but he isn't right, practically speaking, either.
Re: Attention Is Off By One
#24The author identifies a real problem and poses a simple solution. It passes all my crank tests (why did no one come up with this before? Because the author is intimately familiar with the softmax function from work outside of ML, and plausibly nobody who’s investigating these issues is remotely as familiar, so despite researchers narrowing the issue down to “something to do with softmax”, they don’t have a deep enough understanding of softmax to see what’s wrong).
If the author is reading any of these comments, though, I would urge them to expand on their claim that “I’m 99.44% sure that it will resolve the outlier feedback loop”. As it stands, that’s the only explanation we get of how the outliers might be related to softmax!
Re: Attention Is Off By One
#25TL;DR: The author proposes that instead of using the Softmax function in each head, Softmax(x_i) = exp(x_i) / sum(exp(x_i)), we should use instead what the author calls the Softmax_1 function, Softmax_1(x_i) = exp(x_i) / (1 + sum(exp(x_i))), which would make it possible for each transformer head's attention probabilities to be zero, i.e., attend to nothing, by computing x_i's with values well below zero. Giving each…
I also saw the author distinguished internal versus output softmax. I think he'd apply his modification only to internal softmax and let the external force an output.
Re: Attention Is Off By One
#26Shouldn’t this be called Regularized SoftMax? Adding 1 in the denominator looks a lot like a regularization in other ML contexts.
Re: Attention Is Off By One
#27Earlier quoted context omitted.
I think maybe its because he didn't have experimental results that show that it worked. Not a knock against the author, there are just so many things that seem like good ideas that don't end up working well in practice, a paper like this without results is hard to value.
Then again, if you don't have access to giant compute clusters you can't test this, so it's either a blog post or nothing. I believe the outlier problem that this solves only appears for very large models.
That isn’t true at all. Train a smaller model on a smaller dataset. You can even train on your laptop. It’s definitely feasible. This is just a proof of concept, it doesn’t need to beat state of the art.
Re: Attention Is Off By One
#28I don't really understand the subject matter enough, so I apologize in advance for the meta-comment... The author mentions that he would maybe have written this as a scientific paper: > I tried writing a serious-looking research paper about the bug and my proposed fix, but I lost a series of pitched battles against Pytorch and biblatex, so I figured I’d just write a blog post instead. (History is written by the winne…
> This paper
It's not a paper. It's an idea that sounds plausible, presented in a highly entertaining form.
Re: Attention Is Off By One
#29Isn't this why you have the token, or the token?
Re: Attention Is Off By One
#30I don't really understand the subject matter enough, so I apologize in advance for the meta-comment... The author mentions that he would maybe have written this as a scientific paper: > I tried writing a serious-looking research paper about the bug and my proposed fix, but I lost a series of pitched battles against Pytorch and biblatex, so I figured I’d just write a blog post instead. (History is written by the winne…
Nah, scientific papers are supposed to be precise and technical. This reads like those quite frequent suggestions here of switching all equations in papers to plain English or code: it honestly comes from a place of ignorance, and I say that as basically a layman myself.
What should be encouraged is for academics to blog about their research as well. It would even help when recruiting and onboarding new members. Right now the sociological and economical incentives don't promote this at all.