Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TransMLA: Multi-head latent attention is all you need

arxiv.org

Post reply on HN
Print view View original

11–20 of 38 posts

Re: TransMLA: Multi-head latent attention is all you need

#11

Post by kristopolous »

tankenmate wrote: post #8
octocop wrote: post #7

[flagged]

The title of this paper is a reference to a previous paper titled "Attention Is All You Need"[0][1]. This seminal work described the transformer model that is the basis for almost all LLMs, and is almost certainly the most cited paper on AI even though it was only published in 2017. [0] https://arxiv.org/abs/1706.03762 [1] https://en.wikipedia.org/wiki/Attention_Is_All_You_Need

Right, it's an 8 year old reference that's been made hundreds of times.

People seem to love going to the references graveyard, digging up tired and dead ones and drag them around town hoping everyone thinks they're clever.

Also this was from 3 months ago.

Re: TransMLA: Multi-head latent attention is all you need

#15

Post by netdevphoenix »

tankenmate wrote: post #8
octocop wrote: post #7

[flagged]

The title of this paper is a reference to a previous paper titled "Attention Is All You Need"[0][1]. This seminal work described the transformer model that is the basis for almost all LLMs, and is almost certainly the most cited paper on AI even though it was only published in 2017. [0] https://arxiv.org/abs/1706.03762 [1] https://en.wikipedia.org/wiki/Attention_Is_All_You_Need

Why is this the most cited paper in AI and not the original 1943 paper who started it all?

Re: TransMLA: Multi-head latent attention is all you need

#16

Post by tankenmate »

kristopolous wrote: post #11
tankenmate wrote: post #8

Earlier quoted context omitted.

The title of this paper is a reference to a previous paper titled "Attention Is All You Need"[0][1]. This seminal work described the transformer model that is the basis for almost all LLMs, and is almost certainly the most cited paper on AI even though it was only published in 2017. [0] https://arxiv.org/abs/1706.03762 [1] https://en.wikipedia.org/wiki/Attention_Is_All_You_Need

Right, it's an 8 year old reference that's been made hundreds of times. People seem to love going to the references graveyard, digging up tired and dead ones and drag them around town hoping everyone thinks they're clever. Also this was from 3 months ago.

By that argument you must also hate anything that mentions the term "considered harmful", or makes any form of derivative cultural reference (like just about every episode of the Simpsons). Why do you let it get to you?

Re: TransMLA: Multi-head latent attention is all you need

#17

Post by zaptrem »

netdevphoenix wrote: post #15
tankenmate wrote: post #8

Earlier quoted context omitted.

The title of this paper is a reference to a previous paper titled "Attention Is All You Need"[0][1]. This seminal work described the transformer model that is the basis for almost all LLMs, and is almost certainly the most cited paper on AI even though it was only published in 2017. [0] https://arxiv.org/abs/1706.03762 [1] https://en.wikipedia.org/wiki/Attention_Is_All_You_Need

Why is this the most cited paper in AI and not the original 1943 paper who started it all?

Transformers are what made ML infinitely scalable and caused a huge amount of progress in very few years since everyone could just go scale things. However, idk how many of those papers actually even cite the transformer paper?

Re: TransMLA: Multi-head latent attention is all you need

#18

Post by jbellis »

[abstract] This approach significantly reduces the KV cache size relative to traditional multi-head attention

[3.3] For saving the KV cache, only the intermediate latent representations need to be stored: [latex] where r is much smaller than nh · dh [n-sub-h, d-sub-h]

[background] In traditional multi-head attention you must cache full key and value matrices of size T x (nh · dh) where T is the token length, nh is the number of attention heads, dh is the dimensionality of each individual head

sounds like a big win for memory constrained environments like local inference

Re: TransMLA: Multi-head latent attention is all you need

#19

Post by tankenmate »

netdevphoenix wrote: post #15
tankenmate wrote: post #8

Earlier quoted context omitted.

The title of this paper is a reference to a previous paper titled "Attention Is All You Need"[0][1]. This seminal work described the transformer model that is the basis for almost all LLMs, and is almost certainly the most cited paper on AI even though it was only published in 2017. [0] https://arxiv.org/abs/1706.03762 [1] https://en.wikipedia.org/wiki/Attention_Is_All_You_Need

Why is this the most cited paper in AI and not the original 1943 paper who started it all?

Probably because of the modern "publish or perish" mantra led to an exponential growth in publications, and "newer is better" means that newer impactful papers get cited more than older impactful publications. But that thesis is probably a paper in itself (of the meta analysis navel gazing variety).

Re: TransMLA: Multi-head latent attention is all you need

#20

Post by nihzm »

kristopolous wrote: post #11
tankenmate wrote: post #8

Earlier quoted context omitted.

The title of this paper is a reference to a previous paper titled "Attention Is All You Need"[0][1]. This seminal work described the transformer model that is the basis for almost all LLMs, and is almost certainly the most cited paper on AI even though it was only published in 2017. [0] https://arxiv.org/abs/1706.03762 [1] https://en.wikipedia.org/wiki/Attention_Is_All_You_Need

Right, it's an 8 year old reference that's been made hundreds of times. People seem to love going to the references graveyard, digging up tired and dead ones and drag them around town hoping everyone thinks they're clever. Also this was from 3 months ago.

It has definitely been overused by too many authors. This reminds me a passage of Orwell's essay "Politics and the English Language":

> A newly−invented metaphor assists thought by evoking a visual image, while on the other hand a metaphor which is technically "dead" (e.g., iron resolution) has in effect reverted to being an ordinary word and can generally be used without loss of vividness. But in between these two classes there is a huge dump of worn−out metaphors which have lost all evocative power and are merely used because they save people the trouble of inventing phrases for themselves

Post reply on HN