[flagged]
All you need titles stopping is all you need.
They won't stop until someone publishes gotos considered all you need.
TransMLA: Multi-head latent attention is all you need
31–38 of 38 posts
Re: TransMLA: Multi-head latent attention is all you need
#32[flagged]
It's better than "omg, this random thing we tried actually works!"
Re: TransMLA: Multi-head latent attention is all you need
#33Earlier quoted context omitted.
By that argument you must also hate anything that mentions the term "considered harmful", or makes any form of derivative cultural reference (like just about every episode of the Simpsons). Why do you let it get to you?
Because attention and time are a quantity that only decreases with life.
Then why waste your time with getting upset about people making tired cultural references? It's a chuckle at best and a meh at worst, getting bothered by it is a waste of effort.
Re: TransMLA: Multi-head latent attention is all you need
#34Earlier quoted context omitted.
The title of this paper is a reference to a previous paper titled "Attention Is All You Need"[0][1]. This seminal work described the transformer model that is the basis for almost all LLMs, and is almost certainly the most cited paper on AI even though it was only published in 2017. [0] https://arxiv.org/abs/1706.03762 [1] https://en.wikipedia.org/wiki/Attention_Is_All_You_Need
Why is this the most cited paper in AI and not the original 1943 paper who started it all?
Because the MCP neuron is taken as common knowledge and people do not feel the need to explicitly reference it (and haven't for some time), and the pace of publishing has increased in recent years.
Re: TransMLA: Multi-head latent attention is all you need
#35I'm not "in the field" though I like to read about and use LLMs. This video "How DeepSeek Rewrote the Transformer [MLA]"[0] is really good at explaining MHA, MQA, GQA, and MLA with clear visuals/animations and how DeepSeek MLA is 57x more efficient.
Re: TransMLA: Multi-head latent attention is all you need
#36[deleted]
Re: TransMLA: Multi-head latent attention is all you need
#37Very cool idea. Can't wait for converted models on HF.
deepseek-v2,v3,r1 are all using multi-headed attention.
Re: TransMLA: Multi-head latent attention is all you need
#38My (possibly wrong) TLDR: TransMLA is a method to "compress" an already trained GQA model, with the additional option to further fine tune it. Shall make inference faster.
It is not a method to compress a Grouped-Query Attention model, but to expand it into an equivalent Multi-head Latent Attention model with the same key-value cache size but larger effective key/value vectors and a correspondingly larger number of trainable parameters. With additional training, you can then obtain a better model that only uses a little bit more memory.
Thanks for the clarification.