Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Do transformers need three projections? Systematic study of QKV variants

arxiv.org

Post reply on HN
Print view View original

21–30 of 54 posts

Re: Do transformers need three projections? Systematic study of QKV variants

#21

Post by kanbankaren »

amluto wrote: post #5

Hint for authors: when discussing linear algebra (or really most other kinds of math), follow normal conventions. In this case, the convention would be that - (the minus sign) means subtraction. It does not mean "and also", especially when you sandwich it between two variables that represent matrices. I read the paper with much head scratching all the way through sections 1 and 2 and part of 3 before I figured out th…

It confused me too.

A n-tuple notation would have been more readable and mathematically accurate like (Q=K, V), (Q, K=V), and (Q=K=V).

Re: Do transformers need three projections? Systematic study of QKV variants

#22

Post by janalsncm »

in-silico wrote: post #6

These types of ablation studies are always good. However, I'm not sure how generalizable the language model findings here are. Their 1.2B model was trained on only 10B tokens, which is less than half of the chinchilla compute optimal number. Modern overtrained 1B LLMs are trained on the order of 10T tokens (1000x more). This is important because, from my own experience, simplifications and alternatives to standard at…

It’s a data point. I could imagine in a hardware constrained setting we might not care about training on enormous token counts, and on smaller devices it’s great if we can simplify the architecture.

I agree that this isn’t proof that it scales to trillions of tokens, but this does show a scaled up experiment would be worth a shot.

Re: Do transformers need three projections? Systematic study of QKV variants

#23

Post by semiinfinitely »

amluto wrote: post #5

Hint for authors: when discussing linear algebra (or really most other kinds of math), follow normal conventions. In this case, the convention would be that - (the minus sign) means subtraction. It does not mean "and also", especially when you sandwich it between two variables that represent matrices. I read the paper with much head scratching all the way through sections 1 and 2 and part of 3 before I figured out th…

Its not a math paper

Re: Do transformers need three projections? Systematic study of QKV variants

#26

Post by Philpax »

janalsncm wrote: post #22
in-silico wrote: post #6

These types of ablation studies are always good. However, I'm not sure how generalizable the language model findings here are. Their 1.2B model was trained on only 10B tokens, which is less than half of the chinchilla compute optimal number. Modern overtrained 1B LLMs are trained on the order of 10T tokens (1000x more). This is important because, from my own experience, simplifications and alternatives to standard at…

It’s a data point. I could imagine in a hardware constrained setting we might not care about training on enormous token counts, and on smaller devices it’s great if we can simplify the architecture. I agree that this isn’t proof that it scales to trillions of tokens, but this does show a scaled up experiment would be worth a shot.

The Chinchilla scaling laws give you a minimum for the number of tokens you should be using for a given size: if you can't meet what they suggest for that size, you should shrink the size, as, otherwise, the capacity of the model is going to waste.

I do agree that it is a datapoint, but GP's point is that this model was undertrained, so it's hard to draw the same conclusions from it that we would from other research.

Re: Do transformers need three projections? Systematic study of QKV variants

#27

Post by canjobear »

amluto wrote: post #5

Hint for authors: when discussing linear algebra (or really most other kinds of math), follow normal conventions. In this case, the convention would be that - (the minus sign) means subtraction. It does not mean "and also", especially when you sandwich it between two variables that represent matrices. I read the paper with much head scratching all the way through sections 1 and 2 and part of 3 before I figured out th…

It’s not typeset in math mode so you can’t expect the hyphen to correspond to minus.

Re: Do transformers need three projections? Systematic study of QKV variants

#28

Post by volemo »

semiinfinitely wrote: post #23
amluto wrote: post #5

Hint for authors: when discussing linear algebra (or really most other kinds of math), follow normal conventions. In this case, the convention would be that - (the minus sign) means subtraction. It does not mean "and also", especially when you sandwich it between two variables that represent matrices. I read the paper with much head scratching all the way through sections 1 and 2 and part of 3 before I figured out th…

Its not a math paper

Does it not being an English philology paper mean they are free to spell “fish” as “ghoti”?

Re: Do transformers need three projections? Systematic study of QKV variants

#29

Post by sfink »

amluto wrote: post #5

Hint for authors: when discussing linear algebra (or really most other kinds of math), follow normal conventions. In this case, the convention would be that - (the minus sign) means subtraction. It does not mean "and also", especially when you sandwich it between two variables that represent matrices. I read the paper with much head scratching all the way through sections 1 and 2 and part of 3 before I figured out th…

Wha? Why didn't they use Q=K=V for that?

Re: Do transformers need three projections? Systematic study of QKV variants

#30

Post by nullpoint420 »

Lerc wrote: post #11

I can see why the QKV gets used but I can't help but think that thete's got to be a better mechanism with turning a pair of vectors into a new vector and a significance field. Geometrically I imagine the process of attention like picking up a bunch of vectots and spinning and squishing them in many-D until you can find a crack where you can see all the way through, then leveraging that crack to seperate what you want…

It kinda reminds me of general relativity and gravity bending space-time. I'm sure I sound nuts right now, but the model fits in my head.
Post reply on HN