Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Differential Transformer

arxiv.org

Post reply on HN
Print view View original

171–180 of 188 posts

Re: Differential Transformer

#171

Post by physicsguy »

godelski wrote: post #158
Kubuxu wrote: post #102

Earlier quoted context omitted.

Hmm, 0.8 works well, but let's try setting lower layers to lower initial value. Let's say 0.2. Ok, I need a formula that will go between 0.2 and 0.8, slowly approaching 0.8. Starts fiddling with numbers for 20min, I guess this can work.

Sure, but in research show some comparisons

In practice there's always a trade off between getting some result out and published and rigorously exploring every avenue of optimisation in research. Sometimes you have to say 'this is good enough and long enough already'.

Re: Differential Transformer

#172

Post by aDyslecticCrow »

f38zf5vdt wrote: post #155
vessenes wrote: post #140

Earlier quoted context omitted.

Yes. This looks really, really good to me. Cross the board improvements in training time, perplexity improvements per both token trained and per model size. I'm reminded of MoE architectures, in that world we're choosing an optimal small model to process part or all of the inference job; I wonder if MoE got some of the same benefits from forcing the Transformer to distinguish between alternate possibilities. In any e…

There is a downside, every attention layer has to effectively compute attention twice (run scaled_dot_product_attention). As scaled_dot_product_attention is usually one of the most expensive operations in training and inference of a model, it seems like networks using this may be substantially slow and perhaps should considered against larger networks with more attention layers. https://github.com/microsoft/unilm/blo…

In Big-O notation, O(2n) = O(n). Two times slower is actually not that much. If this slowdown results in better inference in the same number of training rounds or better-tuned weights with fewer redundant features, that can be a very worthwhile sacrifice.

It's also a complex optimization problem, not just about computing. Two times, the parameters take more than two times the time to tune and two times the working memory to train and use. There are also plenty of model training scenarios where data throughput from the dataset into memory and back out is the final bottleneck.

So, though I agree it is indeed a downside, I think it's a worthwhile sacrifice if the results they show are reproducible.

Re: Differential Transformer

#173

Post by godelski »

physicsguy wrote: post #171
godelski wrote: post #158

Earlier quoted context omitted.

Sure, but in research show some comparisons

In practice there's always a trade off between getting some result out and published and rigorously exploring every avenue of optimisation in research. Sometimes you have to say 'this is good enough and long enough already'.

Right, but it has to be *good enough*. My issue isn't that they didn't do more work, my issue is that they didn't even report work that they did do that communicates the impact of the literal proposed method.

https://news.ycombinator.com/item?id=41783013

Re: Differential Transformer

#174

Post by dartos »

radarsat1 wrote: post #170
dartos wrote: post #70

Earlier quoted context omitted.

> Hallucination and creativity feel very related to me. Why? I see them as just sampling errors. Sure a mistake can spark inspiration sometimes, but creativity is much more than mistakes. > I understand hallucinating as simply being misaligned with the space humans feel appropriate to interpolate between These language models are next-token predictors. The way the next token is predicted is by sampling a probability…

Often see this argument but it doesn't hold water for me. What we call hallucination is usually when the model says something confidently wrong. Yes the sampling procedure is nondeterministic but this is unrelated to hallucinations. The model can generate a distribution to sample with very little weight on the "wrong" output and then this should be ignored by procedures like top-k sampling. The fact that this doesn't…

> What we call hallucination is usually when the model says something confidently wrong

This is a poor definition that only applies to language models trained to be truthful. If you trained a language model to lie, and it told the truth, that would also be a hallucination.

Or if a model was trained to never sound confident, and it made confident, but correct, claims.

My definition is more accurate.

> Yes the sampling procedure is nondeterministic but this is unrelated to hallucinations.

It’s not the only factor, but it’s absolutely related. It’s also really easy to explain in a comment.

For example, if you always sampled the lowest ranked token, the model would always hallucinate (by output mostly garbage)

Top-k sampling doesn’t eliminate all errors, unless you’re just always picking the most likely token. At that point the sampling process is deterministic, but we’ve seen model output be poor with that setting for reasons I explain next.

> that hallucination is a deeper problem

Of course, it’s because the training process itself is nondeterministic. We can’t make a perfect model, it’s just not how statistical models work.

Re: Differential Transformer

#176

Post by anon291 »

viraptor wrote: post #30
anon291 wrote: post #21

Earlier quoted context omitted.

You need to separate out the LLM, which only produces a set of probabilities, from the system, which includes the LLM and the sampling methodology. Sampling is currently not very intelligent at all. The next bit of confusion is that the 'probability' isn't 'real'. It's not an actual probability but a weight that sums up to one, which is close enough to how probability works that we call it that. However, sometimes th…

> However, sometimes there are several good answers and so all the good answers get a lower probability because there are 5 of them. That's the result after softmax. If you want to act on the raw results, you can still do that.

The results before softmax don't sum to one so don't even act like a probability distribution. And that's the point. When you have the pre-softmax activations, there are infinitely many ways to convert them to something probability-like. You can normalize them after taking the square root, the square, raising to three, etc. Or you can exponentiate and for some reason that does better. Either way it's not a 'real' probability distribution.

Re: Differential Transformer

#177

Post by tananan »

godelski wrote: post #157
Grosvenor wrote: post #139

Earlier quoted context omitted.

Regular softmax (and attention) has an error in it. softmax should be exp()/1+∑exp() Notice the 1 added to the denominator. The difference is at the negative limit, softmax can be 0, instead of some epsilon. The same could be done by adding an extra zero value in x. Downside is, you have to retrain your model from scratch to fix this.

> softmax should be exp()/1+∑exp() You referring to Miller's blogpost?[0] There's not an error in attention. Adding the +1 actually makes it not attention because you no longer generate a probability distribution[1]. There's nothing really preventing attention to have a zero in any of the entries, the thing is that you probably won't get -inf (very large negative number) inside inner product and you're going to have…

Haven't gone through the paper fully, but just looking at the functional form of their attention, it seems more like a constraint on a standard MHA than an architectural discovery.

Take a vanilla MHA, tie the V projection between consecutive heads, make the output projection subtract consecutive heads, with some fixed prefactor and voila, you're most if not all of the way there.

Re: Differential Transformer

#178

Post by cuteboy19 »

aDyslecticCrow wrote: post #172
f38zf5vdt wrote: post #155

Earlier quoted context omitted.

There is a downside, every attention layer has to effectively compute attention twice (run scaled_dot_product_attention). As scaled_dot_product_attention is usually one of the most expensive operations in training and inference of a model, it seems like networks using this may be substantially slow and perhaps should considered against larger networks with more attention layers. https://github.com/microsoft/unilm/blo…

In Big-O notation, O(2n) = O(n). Two times slower is actually not that much. If this slowdown results in better inference in the same number of training rounds or better-tuned weights with fewer redundant features, that can be a very worthwhile sacrifice. It's also a complex optimization problem, not just about computing. Two times, the parameters take more than two times the time to tune and two times the working me…

The O for any transformer is always quadratic

Re: Differential Transformer

#179

Post by seanhunter »

radarsat1 wrote: post #170
dartos wrote: post #70

Earlier quoted context omitted.

> Hallucination and creativity feel very related to me. Why? I see them as just sampling errors. Sure a mistake can spark inspiration sometimes, but creativity is much more than mistakes. > I understand hallucinating as simply being misaligned with the space humans feel appropriate to interpolate between These language models are next-token predictors. The way the next token is predicted is by sampling a probability…

Often see this argument but it doesn't hold water for me. What we call hallucination is usually when the model says something confidently wrong. Yes the sampling procedure is nondeterministic but this is unrelated to hallucinations. The model can generate a distribution to sample with very little weight on the "wrong" output and then this should be ignored by procedures like top-k sampling. The fact that this doesn't…

Yes exactly. It seems intuitive that the model could generate a better distribution and thus cure hallucination but that doesn't actually match what the model does.

The model doesn't sample a probability distribution of individual "facts"[1] it samples a probability distribution of tokens which are generally parts of words, bits of punctuation etc. That we get "facts" out of it which may even be wrong in the first place is an emergent behaviour because of the attention mechanism.

Totally agree that it's a deeper problem and may be intrinsic to the design of the models and the fact that they are trained on a next word prediction task. Karpathy talks about the models as "dreaming text". In that sense it's not surprising that some of it is whacky. Our dreams are too.

[1] By which I mean atomic things that can be right or wrong

Re: Differential Transformer

#180

Post by timlarshanson »

brrrrrm wrote: post #164
Grosvenor wrote: post #139

Earlier quoted context omitted.

Regular softmax (and attention) has an error in it. softmax should be exp()/1+∑exp() Notice the 1 added to the denominator. The difference is at the negative limit, softmax can be 0, instead of some epsilon. The same could be done by adding an extra zero value in x. Downside is, you have to retrain your model from scratch to fix this.

that silly softmax1 blog post is not worth the read. no one uses it in practice if you think about it, the "escape hatch" is the design of the entire transformer dictionary. if Key/Query attention misaligns with Value's weights, you get a layer head that does not attend to anything...

Yep. From what I've seen, if the head wants to do nothing, it can attend to itself = no inter-token communication.

Still, differential attention is pretty interesting & the benchmarking good, seems worth a try! It's in the same vein as linear or non-softmax attention, which also can work.

Note that there is an error below Eq. 1: W^V should be shape [d_model x d_model] not [d_model, 2*d_model] as in the Q, K matrices.

Idea: why not replace the lambda parameterization between softmax operations with something more general, like a matrix or MLP? E.g: Attention is the affine combination of N softmax attention operations (say, across heads). If the transformer learns an identity matrix here, then you know the original formulation was correct for the data; if it's sparse, these guys were right; if it's something else entirely then who knows...

Post reply on HN