From multi-head to latent attention: The evolution of attention mechanisms • Hacker News Board

Re: From multi-head to latent attention: The evolution of attention mechanisms

#21

Post by ACCount37 » Sat, Aug 30, 2025, 1:26 PM UTC

mxkopy wrote: post #18

treyd wrote: post #16

Earlier quoted context omitted.

Has there been research into some hierarchical attention model that has local attention at the scale of sentences and paragraphs that feeds embeddings up to longer range attention across documents?

There’s the hierarchical reasoning model https://arxiv.org/abs/2506.21734 but it’s very new and largely untested Though honestly I don’t think new neural network architectures are going to get us over this local maximum, I think the next steps forward involve something that’s 1. Non lossy 2. Readily interpretable

By now, I seriously doubt any "readily interpretable" claims.

Nothing about human brain is "readily interpretable", and artificial neural networks - which, unlike brains, can be instrumented and experimented on easily - tend to resist interpretation nonetheless.

If there was an easy to reduce ML to "readily interpretable" representations, someone would have done so already. If there were architectures that perform similarly but are orders of magnitude more interpretable, they will be used, because interpretability is desirable. Instead, we get what we get.

Re: From multi-head to latent attention: The evolution of attention mechanisms

#22

Post by airstrike » Sat, Aug 30, 2025, 1:27 PM UTC

ACCount37 wrote: post #19

sivm wrote: post #12

Earlier quoted context omitted.

Attention is all you need for what we have. But attention is a local heuristic. We have brittle coherence and no global state. I believe we need a paradigm shift in architecture to move forward.

Plenty of "we need a paradigm shift in architecture" going around - and no actual architecture that would beat transformers at their strengths as far as eye can see. I remain highly skeptical. I doubt that transformers are the best architecture possible, but they set a high bar. And it sure seems like people who keep making the suggestion that "transformers aren't the future" aren't good enough to actually clear that…

That logic does not hold.

Being able to provide an immediate replacement is not a requirement to point out limitations in current technology.

Re: From multi-head to latent attention: The evolution of attention mechanisms

#23

Post by lucidrains » Sat, Aug 30, 2025, 1:33 PM UTC

attogram wrote: post #5

"Attention Is All You Need" - I've always wondered if the authors of that paper used such a casual and catchy title because they knew it would be groundbreaking and massively cited in the future....

It is a reference to the beatles song, mainly because Noam Shazeer is a music lover

Re: From multi-head to latent attention: The evolution of attention mechanisms

#24

Post by radarsat1 » Sat, Aug 30, 2025, 2:13 PM UTC

sivm wrote: post #12

attogram wrote: post #5

"Attention Is All You Need" - I've always wondered if the authors of that paper used such a casual and catchy title because they knew it would be groundbreaking and massively cited in the future....

Attention is all you need for what we have. But attention is a local heuristic. We have brittle coherence and no global state. I believe we need a paradigm shift in architecture to move forward.

To be fair it would be a lot easier to iterate on ideas if a single experiment didn't cost thousands of dollars and require such massive data. Things have really gotten to the point that it's just not easy for outsiders to contribute if you're not part of a big company or university, and even then you have to justify the expenditure (risk). Paradigm shifts are hard to come by when there is so much momentum in one direction and trying something different carries significant barriers.

Re: From multi-head to latent attention: The evolution of attention mechanisms

#25

Post by ruuda » Sat, Aug 30, 2025, 2:13 PM UTC

attogram wrote: post #5

"Attention Is All You Need" - I've always wondered if the authors of that paper used such a casual and catchy title because they knew it would be groundbreaking and massively cited in the future....

What about the converse, the paper became some massively influential because of the catchy title? Of course the contents are groundbreaking, but that alone is not enough. A groundbreaking paper that nobody knows about cannot have any impact. Even for research, there is a marketing part to it.

Re: From multi-head to latent attention: The evolution of attention mechanisms

#26

Post by cgearhart » Sat, Aug 30, 2025, 2:51 PM UTC

attogram wrote: post #5

"Attention Is All You Need" - I've always wondered if the authors of that paper used such a casual and catchy title because they knew it would be groundbreaking and massively cited in the future....

The title is a succinct snippet that spoke directly to researchers at the time. The transformer architecture is somewhat obvious (especially in retrospect), but it was still very surprising because no one was really going this direction. They were going many other directions… That’s the point of the title: you don’t need all kinds of complicated systems for NLP to work—“attention is all you need”.

After the success of transfer learning for computer vision in the mid-2010s, it was obvious that NLP needed its own transfer learning approach and AlexNet moment.

Lots of research focus around that time was on recurrent models—because that was the conventional wisdom about how you model sequences. Markov chains had led to vanilla RNNs, LSTMs, GRU, etc., which all seemed tantalizingly promising. (MAMBA fans take note.) Attention mechanisms were even used in recurrent models…but so was everything else.

Then came transformers—mixing all the then-best-practice bits with the heretical idea of just not giving a shit about O(n^2) complexity. The vanilla transformer used an encoder-decoder structure like the best translation models had been doing; it used a stack of identical blocks to nudge the output along through the pipe like ResNet; it was pretrained on a multi-task objective using a large document corpus. But then it jettisoned all the other complexity and just let it all rest on the attention mechanism to capture long range dependencies.

It was immediately thrilling, but it was also completely impractical. (I think the largest model had a 500ish token context limit and bigger than hobbyist GPUs.) So it mostly sat on a shelf while people used other “good enough” models for a few years until the hardware got better and a couple folks proved that it could actually work to run these things at massive scale.

And now here we are.

I think they knew what they were saying at the time, but I don’t think they knew that it would remain true for years.

Re: From multi-head to latent attention: The evolution of attention mechanisms

#27

Post by eldenring » Sat, Aug 30, 2025, 2:51 PM UTC

ruuda wrote: post #25

attogram wrote: post #5

"Attention Is All You Need" - I've always wondered if the authors of that paper used such a casual and catchy title because they knew it would be groundbreaking and massively cited in the future....

What about the converse, the paper became some massively influential because of the catchy title? Of course the contents are groundbreaking, but that alone is not enough. A groundbreaking paper that nobody knows about cannot have any impact. Even for research, there is a marketing part to it.

Huh? of course its enough. Transformers immediately started destroying every single baseline out there. The authors definitely knew it was a very significant discovery beforehand.

Re: From multi-head to latent attention: The evolution of attention mechanisms

#28

Post by soulofmischief » Sat, Aug 30, 2025, 2:52 PM UTC

ruuda wrote: post #25

attogram wrote: post #5

"Attention Is All You Need" - I've always wondered if the authors of that paper used such a casual and catchy title because they knew it would be groundbreaking and massively cited in the future....

What about the converse, the paper became some massively influential because of the catchy title? Of course the contents are groundbreaking, but that alone is not enough. A groundbreaking paper that nobody knows about cannot have any impact. Even for research, there is a marketing part to it.

The paper became massively influential because of its contents, not its catchy title. Scientists do not generally read a paper because if its title, they check the abstract and go from there.

Re: From multi-head to latent attention: The evolution of attention mechanisms

#29

Post by ACCount37 » Sat, Aug 30, 2025, 3:05 PM UTC

airstrike wrote: post #22

ACCount37 wrote: post #19

Earlier quoted context omitted.

Plenty of "we need a paradigm shift in architecture" going around - and no actual architecture that would beat transformers at their strengths as far as eye can see. I remain highly skeptical. I doubt that transformers are the best architecture possible, but they set a high bar. And it sure seems like people who keep making the suggestion that "transformers aren't the future" aren't good enough to actually clear that…

That logic does not hold. Being able to provide an immediate replacement is not a requirement to point out limitations in current technology.

What's the value of "pointing out limitations" if this completely fails to drive any improvements?

If any midwit can say "X is deeply flawed" but no one can put together an Y that would beat X, then clearly, pointing out the flaws was never the bottleneck at all.

Re: From multi-head to latent attention: The evolution of attention mechanisms

#30

Post by microtonal » Sat, Aug 30, 2025, 3:16 PM UTC

cgearhart wrote: post #26

attogram wrote: post #5

"Attention Is All You Need" - I've always wondered if the authors of that paper used such a casual and catchy title because they knew it would be groundbreaking and massively cited in the future....

The title is a succinct snippet that spoke directly to researchers at the time. The transformer architecture is somewhat obvious (especially in retrospect), but it was still very surprising because no one was really going this direction. They were going many other directions… That’s the point of the title: you don’t need all kinds of complicated systems for NLP to work—“attention is all you need”. After the success o…

Lots of research focus around that time was on recurrent models—because that was the conventional wisdom about how you model sequences. Markov chains had led to vanilla RNNs, LSTMs, GRU, etc., which all seemed tantalizingly promising. (MAMBA fans take note.) Attention mechanisms were even used in recurrent models…but so was everything else.

I feel like there is a step missing here...

People were using RNN encoders/decoders for machine translation - the encoder was used to make a representation (fixed-size vector) of the source language sentence, the decoder generated the target language sentence from the source representation.

The issue that people were bumping into is that the fixed-sized vector bottlenecked the encoder/decoder architecture. Representing a variable-length source sentence as a fixed-size vector leads to a loss of information that increases with the source sentence length.

People started adding attention to the decoder as a way to work around this issue. Each decoder step could attend to every token (well, RNN hidden representation) of the source sentence. So, this led to the RNN + attention architecture.

The title 'Attention is all you need' comes from the realization that in this architecture the RNN is not needed, neither for the encoder and decoder. It's a message to the field who was using RNNs + attention (to avoid the bottleneck). Of course, the rest was born from that, encoder-only transformer models like BERT and decoder-only models like current LLMs.