Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

From multi-head to latent attention: The evolution of attention mechanisms

vinithavn.medium.com

Post reply on HN
Print view View original

31–40 of 45 posts

Re: From multi-head to latent attention: The evolution of attention mechanisms

#31

Post by cgearhart »

microtonal wrote: post #30
cgearhart wrote: post #26

Earlier quoted context omitted.

The title is a succinct snippet that spoke directly to researchers at the time. The transformer architecture is somewhat obvious (especially in retrospect), but it was still very surprising because no one was really going this direction. They were going many other directions… That’s the point of the title: you don’t need all kinds of complicated systems for NLP to work—“attention is all you need”. After the success o…

Lots of research focus around that time was on recurrent models—because that was the conventional wisdom about how you model sequences. Markov chains had led to vanilla RNNs, LSTMs, GRU, etc., which all seemed tantalizingly promising. (MAMBA fans take note.) Attention mechanisms were even used in recurrent models…but so was everything else. I feel like there is a step missing here... People were using RNN encoders/de…

This is a fair point and clarification. :-)

Re: From multi-head to latent attention: The evolution of attention mechanisms

#33

Post by scragz »

ACCount37 wrote: post #19
sivm wrote: post #12

Earlier quoted context omitted.

Attention is all you need for what we have. But attention is a local heuristic. We have brittle coherence and no global state. I believe we need a paradigm shift in architecture to move forward.

Plenty of "we need a paradigm shift in architecture" going around - and no actual architecture that would beat transformers at their strengths as far as eye can see. I remain highly skeptical. I doubt that transformers are the best architecture possible, but they set a high bar. And it sure seems like people who keep making the suggestion that "transformers aren't the future" aren't good enough to actually clear that…

what ever happened to Google's Titans?

Re: From multi-head to latent attention: The evolution of attention mechanisms

#34

Post by Mallowram »

eldenring wrote: post #27
ruuda wrote: post #25

Earlier quoted context omitted.

What about the converse, the paper became some massively influential because of the catchy title? Of course the contents are groundbreaking, but that alone is not enough. A groundbreaking paper that nobody knows about cannot have any impact. Even for research, there is a marketing part to it.

Huh? of course its enough. Transformers immediately started destroying every single baseline out there. The authors definitely knew it was a very significant discovery beforehand.

[dead]

Re: From multi-head to latent attention: The evolution of attention mechanisms

#35

Post by zackangelo »

mrtesthah wrote: post #2

Do we know if any of these techniques are actually used in the so-called "frontier" models?

Not quite a frontier model but definitely built by a frontier lab: Grok 2 was recently open sourced and I believe it uses a fairly standard MHA architecture with MoE.

Re: From multi-head to latent attention: The evolution of attention mechanisms

#36

Post by Mallowram »

sivm wrote: post #12
attogram wrote: post #5

"Attention Is All You Need" - I've always wondered if the authors of that paper used such a casual and catchy title because they knew it would be groundbreaking and massively cited in the future....

Attention is all you need for what we have. But attention is a local heuristic. We have brittle coherence and no global state. I believe we need a paradigm shift in architecture to move forward.

[dead]

Re: From multi-head to latent attention: The evolution of attention mechanisms

#37

Post by airstrike »

ACCount37 wrote: post #29
airstrike wrote: post #22

Earlier quoted context omitted.

That logic does not hold. Being able to provide an immediate replacement is not a requirement to point out limitations in current technology.

What's the value of "pointing out limitations" if this completely fails to drive any improvements? If any midwit can say "X is deeply flawed" but no one can put together an Y that would beat X, then clearly, pointing out the flaws was never the bottleneck at all.

I think you don't understand how primary research works. Pointing out flaws helps others think about those flaws.

It's not a linear process so I'm not sure the "bottleneck" analogy holds here.

We're not limited to only talking about "the bottleneck". I think the argument is more that we're very close to optimal results for the current approach/architecture, so getting superior outcomes from AI will actually require meaningfully different approaches.

Re: From multi-head to latent attention: The evolution of attention mechanisms

#38

Post by ACCount37 »

airstrike wrote: post #37
ACCount37 wrote: post #29

Earlier quoted context omitted.

What's the value of "pointing out limitations" if this completely fails to drive any improvements? If any midwit can say "X is deeply flawed" but no one can put together an Y that would beat X, then clearly, pointing out the flaws was never the bottleneck at all.

I think you don't understand how primary research works. Pointing out flaws helps others think about those flaws. It's not a linear process so I'm not sure the "bottleneck" analogy holds here. We're not limited to only talking about "the bottleneck". I think the argument is more that we're very close to optimal results for the current approach/architecture, so getting superior outcomes from AI will actually require m…

Where's that "primary research" you're talking about? I certainly don't see it happening here right now.

My point is: saying "transformers are flawed" is dirt cheap. Coming up with anything less flawed isn't.

Re: From multi-head to latent attention: The evolution of attention mechanisms

#39

Post by yorwba »

radarsat1 wrote: post #24
sivm wrote: post #12

Earlier quoted context omitted.

Attention is all you need for what we have. But attention is a local heuristic. We have brittle coherence and no global state. I believe we need a paradigm shift in architecture to move forward.

To be fair it would be a lot easier to iterate on ideas if a single experiment didn't cost thousands of dollars and require such massive data. Things have really gotten to the point that it's just not easy for outsiders to contribute if you're not part of a big company or university, and even then you have to justify the expenditure (risk). Paradigm shifts are hard to come by when there is so much momentum in one dir…

Plenty of research involves small models trained on small amounts of data. You don't necessarily need to do an internet-scale training run to test a new model architecture, you can just compare it to other models of the same size trained on the same data. For example, small-model speedruns are a thing: https://github.com/KellerJordan/modded-nanogpt

Re: From multi-head to latent attention: The evolution of attention mechanisms

#40

Post by HarHarVeryFunny »

sivm wrote: post #12
attogram wrote: post #5

"Attention Is All You Need" - I've always wondered if the authors of that paper used such a casual and catchy title because they knew it would be groundbreaking and massively cited in the future....

Attention is all you need for what we have. But attention is a local heuristic. We have brittle coherence and no global state. I believe we need a paradigm shift in architecture to move forward.

The Transformer was only ever designed to be a better seq-2-seq architecture, so "all you need" implicitly means "all you need for seq-2-seq" (not all you need for AGI), and was anyways more backwards looking than forwards looking.

The preceding seq-2-seq architectures had been RNN (LSTM) based, then RNN + attention (Bahdanau et al "Jointly Learning to Align & Translate"), with the Transformer "attention is all you need" paper then meaning you can drop use of RNNs altogether and just use attention.

Of course NOT using RNNs was the key motivator behind the new Transformer architecture - not only did you not NEED an RNN, but they explicitly wanted to avoid it since the goal was to support parallel vs sequential processing for better performance on the available highly parallel hardware.

Post reply on HN