I would not be surprised if it turned out the exact attention mechanism does not really matter, similarly to the sigmoid, ReLU, GELU movement, only the speed on calculation - and QKV is pretty good at that on the GPUs.
Do transformers need three projections? Systematic study of QKV variants
31–40 of 54 posts
Re: Do transformers need three projections? Systematic study of QKV variants
#32Hint for authors: when discussing linear algebra (or really most other kinds of math), follow normal conventions. In this case, the convention would be that - (the minus sign) means subtraction. It does not mean "and also", especially when you sandwich it between two variables that represent matrices. I read the paper with much head scratching all the way through sections 1 and 2 and part of 3 before I figured out th…
Its not a math paper
Definitely an applied maths paper given that it has been published under CS/ML and been accepted at ICML.
Re: Do transformers need three projections? Systematic study of QKV variants
#33Hint for authors: when discussing linear algebra (or really most other kinds of math), follow normal conventions. In this case, the convention would be that - (the minus sign) means subtraction. It does not mean "and also", especially when you sandwich it between two variables that represent matrices. I read the paper with much head scratching all the way through sections 1 and 2 and part of 3 before I figured out th…
Would it have killed them to use a comma instead?!
Re: Do transformers need three projections? Systematic study of QKV variants
#34Somewhat relevant is a blog-post that likens attention to kernel smoothing: https://bactra.org/notebooks/nn-attention-and-transformers.h... (as discussed before in https://news.ycombinator.com/item?id=38756888)
Re: Do transformers need three projections? Systematic study of QKV variants
#35I can see why the QKV gets used but I can't help but think that thete's got to be a better mechanism with turning a pair of vectors into a new vector and a significance field. Geometrically I imagine the process of attention like picking up a bunch of vectots and spinning and squishing them in many-D until you can find a crack where you can see all the way through, then leveraging that crack to seperate what you want…
> I can't help but think that thete's got to be a better mechanism
What matters is not how good it is in isolation, but how well it scales to giant datasets and supercomputers. So far attention scales the best. It's the most "brute force"-able mechanism
Re: Do transformers need three projections? Systematic study of QKV variants
#36It's interesting to see people are still experimenting with the core concepts of transformers
Re: Do transformers need three projections? Systematic study of QKV variants
#37I'm terribly sorry, but scaling curves or GTFO. Any random pile of linear algebra works fine-ish at small scales. Very few random piles of linear algebra push the Pareto envelope at large scales.
Not every one can afford millions to publish a paper
Re: Do transformers need three projections? Systematic study of QKV variants
#38I can see why the QKV gets used but I can't help but think that thete's got to be a better mechanism with turning a pair of vectors into a new vector and a significance field. Geometrically I imagine the process of attention like picking up a bunch of vectots and spinning and squishing them in many-D until you can find a crack where you can see all the way through, then leveraging that crack to seperate what you want…
>It's always the trade off of a smart complex operation against an absolute crapload of dumb ones.
You can't make attention more specialized without making it less general, which makes LLMs worse as a universal approximator.
Re: Do transformers need three projections? Systematic study of QKV variants
#39I would not be surprised if it turned out the exact attention mechanism does not really matter, similarly to the sigmoid, ReLU, GELU movement, only the speed on calculation - and QKV is pretty good at that on the GPUs.
This has been my thought for a long time. I think all that matters from attention is that there is crosswise comparison going on.
You need some amount of parallel compute and some amount of global comparison.
And the rest is basically a ways to parameters and scale.
(This is in theory, in practice you can get a lot of small % stability and efficiency improvements that really compound in algorithmic details of model architecture)
Re: Do transformers need three projections? Systematic study of QKV variants
#40Hint for authors: when discussing linear algebra (or really most other kinds of math), follow normal conventions. In this case, the convention would be that - (the minus sign) means subtraction. It does not mean "and also", especially when you sandwich it between two variables that represent matrices. I read the paper with much head scratching all the way through sections 1 and 2 and part of 3 before I figured out th…
It’s not typeset in math mode so you can’t expect the hyphen to correspond to minus.
By this logic a lot of applied maths papers become “does not compile” :D