Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

From multi-head to latent attention: The evolution of attention mechanisms

vinithavn.medium.com

Post reply on HN
Print view View original

11–20 of 45 posts

Re: From multi-head to latent attention: The evolution of attention mechanisms

#11

Post by slickytail »

attogram wrote: post #5

"Attention Is All You Need" - I've always wondered if the authors of that paper used such a casual and catchy title because they knew it would be groundbreaking and massively cited in the future....

The transformer was a major breakthrough in NLP, and it was clear at the time of publishing that it would have a major impact. But I will add that it is common in the Deep Learning field to give papers catchy titles (see, off the top of my head: all the YOLO papers, ViT, DiT, textual inversion). The transformer paper is one in a long line of seminal papers with funny names.

Re: From multi-head to latent attention: The evolution of attention mechanisms

#12

Post by sivm »

attogram wrote: post #5

"Attention Is All You Need" - I've always wondered if the authors of that paper used such a casual and catchy title because they knew it would be groundbreaking and massively cited in the future....

Attention is all you need for what we have. But attention is a local heuristic. We have brittle coherence and no global state. I believe we need a paradigm shift in architecture to move forward.

Re: From multi-head to latent attention: The evolution of attention mechanisms

#13

Post by iLoveOncall »

attogram wrote: post #5

"Attention Is All You Need" - I've always wondered if the authors of that paper used such a casual and catchy title because they knew it would be groundbreaking and massively cited in the future....

I recommend reading this article which explains how you can get your papers accepted, and explains that a catchy title is the #1 most important thing: https://maxwellforbes.com/posts/how-to-get-a-paper-accepted/ (not a plug, I just saved it because it was interesting)

Re: From multi-head to latent attention: The evolution of attention mechanisms

#14

Post by iLoveOncall »

qcnguy wrote: post #8
JSR_FDED wrote: post #4

Any way to read this without making an account?

Just click the x at the top right of the interstitial?

That only work for a few articles per month. But usually opening in incognito does the trick.

Re: From multi-head to latent attention: The evolution of attention mechanisms

#15

Post by hyperbovine »

attogram wrote: post #5

"Attention Is All You Need" - I've always wondered if the authors of that paper used such a casual and catchy title because they knew it would be groundbreaking and massively cited in the future....

It sounds like a typical neurips paper to me. And no, they did know what a big deal it would be, else google never would have given the idea away.

Re: From multi-head to latent attention: The evolution of attention mechanisms

#16

Post by treyd »

sivm wrote: post #12
attogram wrote: post #5

"Attention Is All You Need" - I've always wondered if the authors of that paper used such a casual and catchy title because they knew it would be groundbreaking and massively cited in the future....

Attention is all you need for what we have. But attention is a local heuristic. We have brittle coherence and no global state. I believe we need a paradigm shift in architecture to move forward.

Has there been research into some hierarchical attention model that has local attention at the scale of sentences and paragraphs that feeds embeddings up to longer range attention across documents?

Re: From multi-head to latent attention: The evolution of attention mechanisms

#18

Post by mxkopy »

treyd wrote: post #16
sivm wrote: post #12

Earlier quoted context omitted.

Attention is all you need for what we have. But attention is a local heuristic. We have brittle coherence and no global state. I believe we need a paradigm shift in architecture to move forward.

Has there been research into some hierarchical attention model that has local attention at the scale of sentences and paragraphs that feeds embeddings up to longer range attention across documents?

There’s the hierarchical reasoning model https://arxiv.org/abs/2506.21734 but it’s very new and largely untested

Though honestly I don’t think new neural network architectures are going to get us over this local maximum, I think the next steps forward involve something that’s

1. Non lossy

2. Readily interpretable

Re: From multi-head to latent attention: The evolution of attention mechanisms

#19

Post by ACCount37 »

sivm wrote: post #12
attogram wrote: post #5

"Attention Is All You Need" - I've always wondered if the authors of that paper used such a casual and catchy title because they knew it would be groundbreaking and massively cited in the future....

Attention is all you need for what we have. But attention is a local heuristic. We have brittle coherence and no global state. I believe we need a paradigm shift in architecture to move forward.

Plenty of "we need a paradigm shift in architecture" going around - and no actual architecture that would beat transformers at their strengths as far as eye can see.

I remain highly skeptical. I doubt that transformers are the best architecture possible, but they set a high bar. And it sure seems like people who keep making the suggestion that "transformers aren't the future" aren't good enough to actually clear that bar.

Re: From multi-head to latent attention: The evolution of attention mechanisms

#20

Post by miven »

mxkopy wrote: post #18
treyd wrote: post #16

Earlier quoted context omitted.

Has there been research into some hierarchical attention model that has local attention at the scale of sentences and paragraphs that feeds embeddings up to longer range attention across documents?

There’s the hierarchical reasoning model https://arxiv.org/abs/2506.21734 but it’s very new and largely untested Though honestly I don’t think new neural network architectures are going to get us over this local maximum, I think the next steps forward involve something that’s 1. Non lossy 2. Readily interpretable

The ARC Prize Foundation ran extensive ablations on HRM for their slew of reasoning tasks and noted that the "hierarchical" part of their architecture is not much more impactful than a vanilla transformer of the same size with no extra hyperparameter tuning:

https://arcprize.org/blog/hrm-analysis#analyzing-hrms-contri...

Post reply on HN