TransMLA: Multi-head latent attention is all you need
1–10 of 38 posts
Re: TransMLA: Multi-head latent attention is all you need
#2Very cool idea. Can't wait for converted models on HF.
Re: TransMLA: Multi-head latent attention is all you need
#3My (possibly wrong) TLDR: TransMLA is a method to "compress" an already trained GQA model, with the additional option to further fine tune it. Shall make inference faster.
Re: TransMLA: Multi-head latent attention is all you need
#4My (possibly wrong) TLDR: TransMLA is a method to "compress" an already trained GQA model, with the additional option to further fine tune it. Shall make inference faster.
Also makes models smarter ("expressive")
Re: TransMLA: Multi-head latent attention is all you need
#5My (possibly wrong) TLDR: TransMLA is a method to "compress" an already trained GQA model, with the additional option to further fine tune it. Shall make inference faster.
It is not a method to compress a Grouped-Query Attention model, but to expand it into an equivalent Multi-head Latent Attention model with the same key-value cache size but larger effective key/value vectors and a correspondingly larger number of trainable parameters. With additional training, you can then obtain a better model that only uses a little bit more memory.
Re: TransMLA: Multi-head latent attention is all you need
#6[dead]
Re: TransMLA: Multi-head latent attention is all you need
#7[flagged]
Re: TransMLA: Multi-head latent attention is all you need
#8[flagged]
The title of this paper is a reference to a previous paper titled "Attention Is All You Need"[0][1]. This seminal work described the transformer model that is the basis for almost all LLMs, and is almost certainly the most cited paper on AI even though it was only published in 2017.
[0] https://arxiv.org/abs/1706.03762 [1] https://en.wikipedia.org/wiki/Attention_Is_All_You_Need
Re: TransMLA: Multi-head latent attention is all you need
#9Not quite related, but do the mamba models gain ground ?
Answering my own question: https://www.reddit.com/r/MachineLearning/comments/1hpg91o/d_...
Re: TransMLA: Multi-head latent attention is all you need
#10[flagged]
All you need titles stopping is all you need.