[flagged]
I'll give you a call when I finished building my tesla tower. This was also unnoticed by the engineering/science communities.
Transformers Without Normalization
11–20 of 36 posts
Re: Transformers Without Normalization
#12Is it just me or have they provided graphs of LNinput againt LNoutput when the tanh(a*x) is also followed by a weight and bias. Surely you would want to compare the output of the LayerNorm without the weight and bias to get an impression on their similarity. I guess it doesn't matter if the final result works, but I feel like looking at the bit that they are changing in isolation might provide a better insight as to…
From their implementation it looks like they’re calculating tanh and then applying a weight and bias
Exactly, And that's what happens in LayerNorm too. So if figured the best base for comparison would have been to leave that bit out when looking at their difference or similarity, because obviously the bits that have the same implementation will be the same.
Re: Transformers Without Normalization
#13[deleted]
Re: Transformers Without Normalization
#14If true this is very nice incremental improvement. It looks like it doesn't meaningfully improve the capabilities of the model, but is cheaper to compute than RMSNorm (which essentially all current state of art LLMs use) which means faster/cheaper training.
RMSNorm is pretty insigificant in terms of the overall compute in a transformer though -- usually the reduction work can be fused with earlier or later operations.
Rmsnorm acts like a barrier. No compute on the next network layer can start before all compute in the previous layer is done.
Splitting networks across multiple GPU's, this means you must wait for the slowest node and the longest latency.
As soon as you can remove most of these barriers, compute over non-latency-guaranteed networks becomes more practical, as does non-homogeneous compute (ie. Mixing different GPU models).
Re: Transformers Without Normalization
#15When using low precision formats like float8 you usually have to upscale the activations to BF16 before normalising. So the normalisation layers are proportionally using more compute when going to lower precision. Replacing these layers would help reduce the compute cost significantly.
Re: Transformers Without Normalization
#16Earlier quoted context omitted.
RMSNorm is pretty insigificant in terms of the overall compute in a transformer though -- usually the reduction work can be fused with earlier or later operations.
Rmsnorm acts like a barrier. No compute on the next network layer can start before all compute in the previous layer is done. Splitting networks across multiple GPU's, this means you must wait for the slowest node and the longest latency. As soon as you can remove most of these barriers, compute over non-latency-guaranteed networks becomes more practical, as does non-homogeneous compute (ie. Mixing different GPU mode…
What are other barriers in transformers? Or is the normalization layer the primary one?
Re: Transformers Without Normalization
#17Earlier quoted context omitted.
Rmsnorm acts like a barrier. No compute on the next network layer can start before all compute in the previous layer is done. Splitting networks across multiple GPU's, this means you must wait for the slowest node and the longest latency. As soon as you can remove most of these barriers, compute over non-latency-guaranteed networks becomes more practical, as does non-homogeneous compute (ie. Mixing different GPU mode…
What are other barriers in transformers? Or is the normalization layer the primary one?
dot-product attention is the biggest barrier. This is why there are so many attempts to linearize it.
Re: Transformers Without Normalization
#18If true this is very nice incremental improvement. It looks like it doesn't meaningfully improve the capabilities of the model, but is cheaper to compute than RMSNorm (which essentially all current state of art LLMs use) which means faster/cheaper training.
RMSNorm is pretty insigificant in terms of the overall compute in a transformer though -- usually the reduction work can be fused with earlier or later operations.
The paper's Table 7 shows DyT reducing overall LLaMA 7B inference time by 7.8% and training time by 8.2%. That is not insignificant.
Re: Transformers Without Normalization
#19And so vanishing gradients are not a thing anymore?
Good question. That was an issue with tanh as activation function, and before residual connections and normalization layers. Tanh as a normalization but with other activations and residual present apparently is ok.
Re: Transformers Without Normalization
#20And so vanishing gradients are not a thing anymore?
Proper initialization of layers keeps gradient magnitudes from vanishing/exploding in deep networks. If you make sure the output of each layer has mean 0, std 1, the gradients will be reasonable as well, for example.
I recommend e.g. the og resnet paper and its follow-up from Kaiming He et al.
For a modern take on RNNs, read https://arxiv.org/abs/2303.06349 by DeepMind.
There essentially the point is that largest eigenvalue (spectral radius) needs to be around 1, meaning repeated applications of a linear transformation doesn’t cause increase or decrease of the activations.