Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Transformers Without Normalization

jiachenzhu.github.io

Post reply on HN
Print view View original

21–30 of 36 posts

Re: Transformers Without Normalization

#21

Post by tsurba »

imjonse wrote: post #19
blackbear_ wrote: post #8

And so vanishing gradients are not a thing anymore?

Good question. That was an issue with tanh as activation function, and before residual connections and normalization layers. Tanh as a normalization but with other activations and residual present apparently is ok.

Proper initialization is more important.

Batch norm and others are important for faster convergence due to forcing the model to focus creating second and higher order nonlinearities, as a simple shift in mean/std is normalized out, and thus the gradient does not point in a direction that would only change those properties of the output distribution.

Re: Transformers Without Normalization

#22

Post by blackbear_ »

tsurba wrote: post #20
blackbear_ wrote: post #8

And so vanishing gradients are not a thing anymore?

Proper initialization of layers keeps gradient magnitudes from vanishing/exploding in deep networks. If you make sure the output of each layer has mean 0, std 1, the gradients will be reasonable as well, for example. I recommend e.g. the og resnet paper and its follow-up from Kaiming He et al. For a modern take on RNNs, read https://arxiv.org/abs/2303.06349 by DeepMind. There essentially the point is that largest eig…

Sure initialization helps, but are there also results about long term training dynamics? Even the paper you suggested had to use some sort of normalization to keep things stable

Re: Transformers Without Normalization

#23

Post by kouteiheika »

kouteiheika wrote: post #4

If true this is very nice incremental improvement. It looks like it doesn't meaningfully improve the capabilities of the model, but is cheaper to compute than RMSNorm (which essentially all current state of art LLMs use) which means faster/cheaper training.

Okay, I just tried this on my pet transformer training benchmark and the results are very disappointing; it converges much more slowly than just using RMSNorm.

It either needs some significant hyperparameter tuning (besides tweaking alpha, which doesn't seem to do much for me), or some fancier initialization (tried both pytorch default and orthogonal, no difference), or maybe my scalar optimizer doesn't work on it (I have a custom optimizer for scalars which speeds up convergence vs Adam, but for DyT layers it seems to be just as good as Adam), or maybe it only catches up after billions of tokens (which I don't have the budget to test for so long).

Re: Transformers Without Normalization

#25

Post by amitport »

woadwarrior01 wrote: post #17
elcritch wrote: post #16

Earlier quoted context omitted.

What are other barriers in transformers? Or is the normalization layer the primary one?

dot-product attention is the biggest barrier. This is why there are so many attempts to linearize it.

that fail... linearization is a bad idea. But plenty of other optimizations are done

Re: Transformers Without Normalization

#26

Post by kadushka »

kouteiheika wrote: post #23
kouteiheika wrote: post #4

If true this is very nice incremental improvement. It looks like it doesn't meaningfully improve the capabilities of the model, but is cheaper to compute than RMSNorm (which essentially all current state of art LLMs use) which means faster/cheaper training.

Okay, I just tried this on my pet transformer training benchmark and the results are very disappointing; it converges much more slowly than just using RMSNorm. It either needs some significant hyperparameter tuning (besides tweaking alpha, which doesn't seem to do much for me), or some fancier initialization (tried both pytorch default and orthogonal, no difference), or maybe my scalar optimizer doesn't work on it (I…

Which model are you training and on what dataset?

Re: Transformers Without Normalization

#28

Post by Herring »

atgctg wrote: post #18
rryan wrote: post #9

Earlier quoted context omitted.

RMSNorm is pretty insigificant in terms of the overall compute in a transformer though -- usually the reduction work can be fused with earlier or later operations.

The paper's Table 7 shows DyT reducing overall LLaMA 7B inference time by 7.8% and training time by 8.2%. That is not insignificant.

But LLM performance scales according to the log of compute, so yeah it’s pretty insignificant. I think we’ve reached a bit of a plateau.

Re: Transformers Without Normalization

#30

Post by tripplyons »

blackbear_ wrote: post #8

And so vanishing gradients are not a thing anymore?

I think ResNet pretty much solved vanishing gradients. As for exploding gradients, that is typically with good parameter initialization and normalization. The paper in question proposes an alternative to normalization.
Post reply on HN