111–120 of 188 posts
It has to be done in a hierarchical way to know what you attended to + full context.
If the differential vector is being computed with the same input as the attention vector how do you know how to modify the attention vector correctly
Earlier quoted context omitted.
> predict a weight of exactly zero for some of the values Wouldn’t this be pretty unlikely, though?
Quite the opposite — if you have a long sequence only a smattering of the words will influence the meaning of the current word. Everything else is “noise”. Attention is really good at finding this smattering of words (ie assign most weight there). But it struggles to put exactly 0 on the other words.
SoftmaxA[n] - SoftmaxB[n] is exactly 0?
Even if 2 attention layers learn two different things, I would imagine the corresponding weights in each layer wouldn’t exactly cancel each other out.
Like most things in this new world of Machine Learning, I'm really confused why this works? The analogy to noise-cancelling headphones is helpful but in that case we clearly know which is signal and which is noise. Here, if we knew why would we even bother to the noise-cancelling work?
With a single softmax you cannot predict exactly 0, but only very small numbers. When you have a large number of values to add up, this "poisons" the output with a lot of irrelevant stuff (the noise mentioned in the paper). To make things worse, low attention values will have very low gradient, thus needing a lot of weight updates to undo that kind of mistakes. On the other hand, subtracting the output of two softmax…
I wonder if a different approach without that issue exists. For instance, using max(0, exp(x)-1) instead of exp(x) in the softmax attention formula. That way when the query is orthogonal to the key (or worse), it does not contribute.
Earlier quoted context omitted.
> Hallucination and creativity feel very related to me. Why? I see them as just sampling errors. Sure a mistake can spark inspiration sometimes, but creativity is much more than mistakes. > I understand hallucinating as simply being misaligned with the space humans feel appropriate to interpolate between These language models are next-token predictors. The way the next token is predicted is by sampling a probability…
LLM’s are too unpredictable for many practical uses so I’d guess better predictability is better. Hopefully the change the paper proposes will help! But here’s a case for the other side: sure, most mistakes are just errors, but evolution happens via “mistakes.” Also, LLM’s often deliberately add add randomness at inference time.
That’s a nice slogan, but it’s a gross oversimplification.
In the natural world, you can say that mistakes in DNA replication leads to evolution, but that’s discounting the entire process of natural selection.
Same with creativity. Look at Picasso. His was a technically brilliant realistic painter at 15, but his work later in life evolved to be more abstract and weird. I don’t think that was the result of mistakes, but rather intentionally breaking patterns he learned in his youth.
I wonder what is lost here. Surely there's a trade-off... I'm wondering if there's any effect of "creativity", or ability to interpolate between concepts. Hallucination and creativity feel very related to me. I understand hallucinating as simply being misaligned with the space humans feel appropriate to interpolate between
> Hallucination and creativity feel very related to me. Why? I see them as just sampling errors. Sure a mistake can spark inspiration sometimes, but creativity is much more than mistakes. > I understand hallucinating as simply being misaligned with the space humans feel appropriate to interpolate between These language models are next-token predictors. The way the next token is predicted is by sampling a probability…
Exactly. Don't forget that an important factor in the success of GPT3 was RLHF, which is essentially training the model to produce "hallucinations" that are more acceptable on average to human trainers.
Like most things in this new world of Machine Learning, I'm really confused why this works? The analogy to noise-cancelling headphones is helpful but in that case we clearly know which is signal and which is noise. Here, if we knew why would we even bother to the noise-cancelling work?
With a single softmax you cannot predict exactly 0, but only very small numbers. When you have a large number of values to add up, this "poisons" the output with a lot of irrelevant stuff (the noise mentioned in the paper). To make things worse, low attention values will have very low gradient, thus needing a lot of weight updates to undo that kind of mistakes. On the other hand, subtracting the output of two softmax…
If we’re subtracting one attention matrix from another, we’d end up with attention scores between -1 and 1, with a probability of effectively 0 for any single entry to exactly equal 0.
What’s more, the learnable parameter \lambda allows for negative values. This would allow the model to learn to actually add the attention scores, making a score of exactly 0 impossible.
Very clever. I like this kind of nitty-gritty detail work, and the change is small enough to be adapted easily by others. Bravo! I'm a little concerned about the last sentence of the section introduction of "2 Differential Transformer". It mentions using improvements from previous papers, but in the grammatical context, it's unclear if this improvement is added to both the normal transformer and their diff transforme…
The two other changes they mention have been widely adopted, and are included in at least some of the models they benchmark against. It seems they list them for completeness as changes to the original transformer architecture.
I feel like I'm missing a key insight here. I understand the problem that regular softmax attention struggles to approach assigning zero attention to irrelevant stuff. And I get that having this subtraction formula makes it possible to assign exactly (or near) zero attention weight without having crazy outlier activations. But it seems like it also makes it very easy to have negative attention weight (which is equiva…
Like most things in this new world of Machine Learning, I'm really confused why this works? The analogy to noise-cancelling headphones is helpful but in that case we clearly know which is signal and which is noise. Here, if we knew why would we even bother to the noise-cancelling work?
With a single softmax you cannot predict exactly 0, but only very small numbers. When you have a large number of values to add up, this "poisons" the output with a lot of irrelevant stuff (the noise mentioned in the paper). To make things worse, low attention values will have very low gradient, thus needing a lot of weight updates to undo that kind of mistakes. On the other hand, subtracting the output of two softmax…