From multi-head to latent attention: The evolution of attention mechanisms
vinithavn.medium.com
From multi-head to latent attention: The evolution of attention mechanisms
1–10 of 45 posts
Re: From multi-head to latent attention: The evolution of attention mechanisms
#2Do we know if any of these techniques are actually used in the so-called "frontier" models?
Re: From multi-head to latent attention: The evolution of attention mechanisms
#3Do we know if any of these techniques are actually used in the so-called "frontier" models?
The model names are mentioned under each type of attention mechanism
Re: From multi-head to latent attention: The evolution of attention mechanisms
#4Any way to read this without making an account?
Re: From multi-head to latent attention: The evolution of attention mechanisms
#5"Attention Is All You Need" - I've always wondered if the authors of that paper used such a casual and catchy title because they knew it would be groundbreaking and massively cited in the future....
Re: From multi-head to latent attention: The evolution of attention mechanisms
#6Any way to read this without making an account?
Re: From multi-head to latent attention: The evolution of attention mechanisms
#7Any way to read this without making an account?
Re: From multi-head to latent attention: The evolution of attention mechanisms
#8Any way to read this without making an account?
Just click the x at the top right of the interstitial?
Re: From multi-head to latent attention: The evolution of attention mechanisms
#9"Attention Is All You Need" - I've always wondered if the authors of that paper used such a casual and catchy title because they knew it would be groundbreaking and massively cited in the future....
Definitely. I always assumed that, having been involved in writing similarly groundbreaking papers… or so we thought at the time. All my coauthors spent significant time thinking about what the best title would be, and strategies like that were common. (It ended up not mattering for us.)
Re: From multi-head to latent attention: The evolution of attention mechanisms
#10Do we know if any of these techniques are actually used in the so-called "frontier" models?
Who knows what the closed source models use but certainly going by what's happening in open models all the big changes and corresponding gains in capability are in training techniques not model architecture. Things like GQA and MLA as discussed in this article are important techniques for getting better scaling but are relatively minor tweak vs the evolution in training techniques.
I suspect closed models aren't doing anything too radically different from what's presented here.