now if only we could teach humans to also not annotate when they have nothing to add
Attention Is Off By One
41–50 of 347 posts
Re: Attention Is Off By One
#42> their existence is contrary to everything we thought we knew about neural networks prior to building ones that worked so well
Why is this true? Because their existence implies some sort of preferred basis that aligns with the dims of the neural network, which is surprising?
It's not obvious why their existence is so contrary to what we knew.
Re: Attention Is Off By One
#43The author says to add a unity vector to the context, i presume of each layer, to not mess with gradient calculations. But most modern DL frameworks compute the gradient for you, (i know this is true for JAX and Pytorch). Is it maybe that hand coded gradient for a well-known enough dl architecture like transformer is faster than letting the framework autodiff it? Otherwise, i fear some of the 'magic' of transformer n…
think the author is talking about having to fix the extra vector in V to be zeros and making sure to not compute/apply gradients to it
Re: Attention Is Off By One
#44Don't transformers typically have a token at the beginning of the prompt? This seems equivalent to letting the network attend to this token, and produce a zero value if that's what it wants.
Chat-tuned ones do, but the base models don't. For example, Llama doesn't, but Alpaca has "### Instruction:", "### Input:", and "### Response:".
Re: Attention Is Off By One
#45Earlier quoted context omitted.
I was just looking at doing this in pretraining, so I was looking at pretraining losses. The difference was within the range of usual noise so I didn't keep trying.
this is fixing a different issue, not the one you are measuring.
It wasn't really the goal of my experiment to fix this issue for sure, I was trying to see if you could improve attention by decoupling the key used by a position for itself and for future tokens.
Open to being wrong here, but wouldn't it be functionally similar to adding a constant to the softmax denom? the function could sort of learn a specific position to have sink and q multiply to one, then removing it before multipling with v would be exactly identical?
Re: Attention Is Off By One
#46The author identifies a real problem and poses a simple solution. It passes all my crank tests (why did no one come up with this before? Because the author is intimately familiar with the softmax function from work outside of ML, and plausibly nobody who’s investigating these issues is remotely as familiar, so despite researchers narrowing the issue down to “something to do with softmax”, they don’t have a deep enoug…
Yeah, but it lacks the most important test: results. He hasn't actually tried it, he just thinks it will work.
For such a simple change to the softmax it wouldn't take long to verify. It's really embarrassing to not do that before publishing.
Re: Attention Is Off By One
#47The proposed replacement definitely makes more sense (and I've always found the absence of a "failed query" to be puzzling in standard attention), but, in deep learning, things that make more sense don't always actually get better results. So I'm curious whether this has been tried and carefully evaluated.
It would be an amusing find if "Black Swan mega-activations" actually but yet unintentionally made the model smarter...
Re: Attention Is Off By One
#48It would be fun when once in the future someone finds a bug like this, merges the PR, and BAM! Singularity.
Re: Attention Is Off By One
#49I don't really understand the subject matter enough, so I apologize in advance for the meta-comment... The author mentions that he would maybe have written this as a scientific paper: > I tried writing a serious-looking research paper about the bug and my proposed fix, but I lost a series of pitched battles against Pytorch and biblatex, so I figured I’d just write a blog post instead. (History is written by the winne…
Nah, scientific papers are supposed to be precise and technical. This reads like those quite frequent suggestions here of switching all equations in papers to plain English or code: it honestly comes from a place of ignorance, and I say that as basically a layman myself. What should be encouraged is for academics to blog about their research as well. It would even help when recruiting and onboarding new members. Righ…
I agree with everything you say. Though papers really are a bit too hard to read sometimes, but I'd argue it's often not for an overly technical tone so much as writers cutting out a lot of background material for brevity and assumed familiarity.
>What should be encouraged is for academics to blog about their research as well. It would even help when recruiting and onboarding new members. Right now the sociological and economical incentives don't promote this at all.
I will add onto this that a lot of journals have been pushing for video abstracts and "plain English" abstracts. For the most part I don't see these too often but when they're there they're appreciated, and I vaguely recall that someone found that citations go up when they're used (specifically plain English, I don't think anything has been on video abstracts).
There are a lot of good blogs for computational academic subjects (ml, bioinformatics, comp neuro, etc) but I see less for bio and non-software engineering. Math and physics seems to have some really notable blogs, but beyond what gets posted to HN and linked further on those blogs, I can't comment.
Re: Attention Is Off By One
#50>The problem with using softmax is that it forces each attention head to make an annotation, even if it has no information to add to the output vector. Using softmax to choose among discrete alternatives is great; using it for optional annotation (i.e. as input into addition) is, like, not cool, man. The problem here is exacerbated with multi-head attention, as a specialized head is more likely to want to “pass” than a general-purpose one. These attention heads are needlessly noisy, a deafening democracy where abstention is disallowed.
Can't the MLP that processes the concatenated outputs the attention heads handle this? I don't understand why it should be critical that a head be allowed to put something close to zero in its segment of the concatenated vector if it's immediately going to get projected by an MLP anyway.