Transformers Without Normalization
jiachenzhu.github.io
Transformers Without Normalization
1–10 of 36 posts
Re: Transformers Without Normalization
#2What are the practical implications of this?
Re: Transformers Without Normalization
#3What are the practical implications of this?
from the abstract
By incorporating DyT, Transformers without normalization can match or exceed the performance of their normalized counterparts, mostly without hyperparameter tuning.Re: Transformers Without Normalization
#4If true this is very nice incremental improvement. It looks like it doesn't meaningfully improve the capabilities of the model, but is cheaper to compute than RMSNorm (which essentially all current state of art LLMs use) which means faster/cheaper training.
Re: Transformers Without Normalization
#5[flagged]
Re: Transformers Without Normalization
#6Need to read the details, but removing the norm can be big. It’s always a pain to make sure that your network is normalized properly when trying new architectures. Likely there will still be other implications of the tanh, since the norm is sometimes solving a conditioning problem, but IMO more alternatives are welcome
Re: Transformers Without Normalization
#7Is it just me or have they provided graphs of LNinput againt LNoutput when the tanh(a*x) is also followed by a weight and bias.
Surely you would want to compare the output of the LayerNorm without the weight and bias to get an impression on their similarity.
I guess it doesn't matter if the final result works, but I feel like looking at the bit that they are changing in isolation might provide a better insight as to what is happening.
Re: Transformers Without Normalization
#8And so vanishing gradients are not a thing anymore?
Re: Transformers Without Normalization
#9If true this is very nice incremental improvement. It looks like it doesn't meaningfully improve the capabilities of the model, but is cheaper to compute than RMSNorm (which essentially all current state of art LLMs use) which means faster/cheaper training.
RMSNorm is pretty insigificant in terms of the overall compute in a transformer though -- usually the reduction work can be fused with earlier or later operations.
Re: Transformers Without Normalization
#10Is it just me or have they provided graphs of LNinput againt LNoutput when the tanh(a*x) is also followed by a weight and bias. Surely you would want to compare the output of the LayerNorm without the weight and bias to get an impression on their similarity. I guess it doesn't matter if the final result works, but I feel like looking at the bit that they are changing in isolation might provide a better insight as to…
From their implementation it looks like they’re calculating tanh and then applying a weight and bias