Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Do transformers need three projections? Systematic study of QKV variants

arxiv.org

Post reply on HN
Print view View original

11–20 of 54 posts

Re: Do transformers need three projections? Systematic study of QKV variants

#11

Post by Lerc »

I can see why the QKV gets used but I can't help but think that thete's got to be a better mechanism with turning a pair of vectors into a new vector and a significance field.

Geometrically I imagine the process of attention like picking up a bunch of vectots and spinning and squishing them in many-D until you can find a crack where you can see all the way through, then leveraging that crack to seperate what you want.

I doubt that's strictly accurate, but it might be close enough that it makes me think that if you were doing that with a bunch of bananas, it would be much easier to find the way through if you could also bend the bunch so they were all straight.

It's always the trade off of a smart complex operation against an absolute crapload of dumb ones.

Re: Do transformers need three projections? Systematic study of QKV variants

#13

Post by ketchup32613 »

jephs wrote: post #10

I'm terribly sorry, but scaling curves or GTFO. Any random pile of linear algebra works fine-ish at small scales. Very few random piles of linear algebra push the Pareto envelope at large scales.

Do you want to see scaling curves wrt data and param size? I agree that 1.2B and 10B tokens is not representative, but what scale of parameters and dataset sizes would be convincing?

Re: Do transformers need three projections? Systematic study of QKV variants

#14

Post by Hugsun »

brianjmingus wrote: post #12

[dead]

Is there anything of value in this project?

It sounds interesting at a glance, but it seems to be AI slop. So it's hard to tell if there are any interesting discoveries there, or just some worthless results described with performatively advanced language.

Re: Do transformers need three projections? Systematic study of QKV variants

#15

Post by zxexz »

ketchup32613 wrote: post #13
jephs wrote: post #10

I'm terribly sorry, but scaling curves or GTFO. Any random pile of linear algebra works fine-ish at small scales. Very few random piles of linear algebra push the Pareto envelope at large scales.

Do you want to see scaling curves wrt data and param size? I agree that 1.2B and 10B tokens is not representative, but what scale of parameters and dataset sizes would be convincing?

Not to sound facetious, but perhaps enough runs at different param/token sizings to define a curve?

Re: Do transformers need three projections? Systematic study of QKV variants

#17

Post by [deleted] »

Hugsun wrote: post #14
brianjmingus wrote: post #12

[dead]

Is there anything of value in this project? It sounds interesting at a glance, but it seems to be AI slop. So it's hard to tell if there are any interesting discoveries there, or just some worthless results described with performatively advanced language.

[deleted]

Re: Do transformers need three projections? Systematic study of QKV variants

#20

Post by ACCount37 »

in-silico wrote: post #6

These types of ablation studies are always good. However, I'm not sure how generalizable the language model findings here are. Their 1.2B model was trained on only 10B tokens, which is less than half of the chinchilla compute optimal number. Modern overtrained 1B LLMs are trained on the order of 10T tokens (1000x more). This is important because, from my own experience, simplifications and alternatives to standard at…

I wonder if some of those synthetics that specifically burn in attention inductive bias could help there - i.e. by getting attention to converge faster than it normally would?
Post reply on HN