Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Do transformers need three projections? Systematic study of QKV variants

arxiv.org

Post reply on HN
Print view View original

41–50 of 54 posts

Re: Do transformers need three projections? Systematic study of QKV variants

#41

Post by simsla »

sfink wrote: post #29
amluto wrote: post #5

Hint for authors: when discussing linear algebra (or really most other kinds of math), follow normal conventions. In this case, the convention would be that - (the minus sign) means subtraction. It does not mean "and also", especially when you sandwich it between two variables that represent matrices. I read the paper with much head scratching all the way through sections 1 and 2 and part of 3 before I figured out th…

Wha? Why didn't they use Q=K=V for that?

The notation is supposed to mean: you have a matrix Q, and also a shared K=V matrix.

I agree with GP that it's super confusing to us the minus sign as a delimiter between formulas. The tuple notation suggested elsewhere would be way clearer.

Re: Do transformers need three projections? Systematic study of QKV variants

#42

Post by Sharlin »

canjobear wrote: post #27
amluto wrote: post #5

Hint for authors: when discussing linear algebra (or really most other kinds of math), follow normal conventions. In this case, the convention would be that - (the minus sign) means subtraction. It does not mean "and also", especially when you sandwich it between two variables that represent matrices. I read the paper with much head scratching all the way through sections 1 and 2 and part of 3 before I figured out th…

It’s not typeset in math mode so you can’t expect the hyphen to correspond to minus.

Cannot tell whether sarcasm or not.

Re: Do transformers need three projections? Systematic study of QKV variants

#44

Post by Der_Einzige »

jephs wrote: post #10

I'm terribly sorry, but scaling curves or GTFO. Any random pile of linear algebra works fine-ish at small scales. Very few random piles of linear algebra push the Pareto envelope at large scales.

This exact mentality is cancer for peer review/the industry. We all know who you are if you are using 1000+ TPUs, and yes you do get a "buff" to your peer review scores because people know where you work.

Fuck your scaling curves. More research labs need to #yolo and try stuff that doesn't have good scaling behavior proven yet. State Space models have continued to take forever to proliferate despite being objectively good because only the god dang Chinese understand that you actually need to #yolo sometimes like making some of your layer state space layers in Hunyuan-T1.

Re: Do transformers need three projections? Systematic study of QKV variants

#46

Post by jephs »

Der_Einzige wrote: post #44
jephs wrote: post #10

I'm terribly sorry, but scaling curves or GTFO. Any random pile of linear algebra works fine-ish at small scales. Very few random piles of linear algebra push the Pareto envelope at large scales.

This exact mentality is cancer for peer review/the industry. We all know who you are if you are using 1000+ TPUs, and yes you do get a "buff" to your peer review scores because people know where you work. Fuck your scaling curves. More research labs need to #yolo and try stuff that doesn't have good scaling behavior proven yet. State Space models have continued to take forever to proliferate despite being objectively…

Scaling curves don't need to be drawn at particularly enormous parameter counts to be useful! If you can do a 300M and 1.2B run (like the authors do here), then you can do 150M, 300M, 600M, and 1.2B runs with only 50% more resources, and get a much better sense for whether effects seem to amplify or diminish as scale increases.

Re: Do transformers need three projections? Systematic study of QKV variants

#47

Post by mattalex »

soVeryTired wrote: post #45

Can anyone explain to me why Q and K are both needed? They only ever appear as a pair, so why can’t you just define a matrix A = QK and learn that directly?

Because the size of the attention matrix depends on the number of tokens (this is what makes attention N^2). If you don't care about having a flexible number of input tokens (e.g. in image processing) you can learn a fixed routing matrix. This is known as an MLP mixer https://arxiv.org/pdf/2105.01601 : you have one layer that processes each token in isolation ("vertical MLP") but ignores the inter-token connections, followed by a layer that combines between tokens ("horizontal MLP") that treats the internals of every token identically.

Re: Do transformers need three projections? Systematic study of QKV variants

#48

Post by spindump8930 »

WithinReason wrote: post #37
jephs wrote: post #10

I'm terribly sorry, but scaling curves or GTFO. Any random pile of linear algebra works fine-ish at small scales. Very few random piles of linear algebra push the Pareto envelope at large scales.

Not every one can afford millions to publish a paper

That's why you do several small and medium scale tests, fit a curve, and ideally show that the trend persists at several scales. Not a single large or medium run - see the other comments down thread for example sizes.

Re: Do transformers need three projections? Systematic study of QKV variants

#49

Post by spindump8930 »

7e wrote: post #18

More evidence that the original Transformer authors didn't really know what they were doing, but they did have access to more cheap compute than anyone else.

Can you share the specific part of this work that demonstrates better scaling than original transformers? Also note that many of the changes to that architecture, that have been proven in their use at actual scale, were brought about by members of the original team. Most notably Noam Shazeer.

Re: Do transformers need three projections? Systematic study of QKV variants

#50

Post by spindump8930 »

jephs wrote: post #46
Der_Einzige wrote: post #44

Earlier quoted context omitted.

This exact mentality is cancer for peer review/the industry. We all know who you are if you are using 1000+ TPUs, and yes you do get a "buff" to your peer review scores because people know where you work. Fuck your scaling curves. More research labs need to #yolo and try stuff that doesn't have good scaling behavior proven yet. State Space models have continued to take forever to proliferate despite being objectively…

Scaling curves don't need to be drawn at particularly enormous parameter counts to be useful! If you can do a 300M and 1.2B run (like the authors do here), then you can do 150M, 300M, 600M, and 1.2B runs with only 50% more resources, and get a much better sense for whether effects seem to amplify or diminish as scale increases.

Exactly. Good peer reviewers understand that you can also move down on the scaling curve, not just up. Also laughable to try a "yolo" run without validating a scaling ladder/curve.
Post reply on HN