The toggle is really useful. Liked it!!
A walk through of the DeltaNet family of linear attention variants
11–20 of 134 posts
Re: A walk through of the DeltaNet family of linear attention variants
#12You know its a doozy when the author writes a disclaimer at the top saying that bra-ket notation was chosen in order to make the algorithm and data structures clearer.
One of the more annoying parts of my physics study was getting used to the new matrix multiplication notation they came up with every semester.
Re: A walk through of the DeltaNet family of linear attention variants
#13No, you couldn't have. There are plenty of ML innovations that when push comes to shove only depend on having access to more compute, but this is one of the worst examples I've ever seen.
I always thought that the jump from LSTM/GRU -> Attention wasn't a particularly big one. Instead of partial unroll, do a full unroll. Why not (because it's too expensive, that's why not). Every component was known, and everybody anywhere near ML knew perfectly well why NOT to try that: because you just don't have the compute to fully unroll an LSTM. From that point attention is optimized (they key-query mechanic). The big innovation is not so much the mechanism itself but realizing the parallelize-ability of it.
It's sort of like if one would today make the "improvement" to attention to replace they key-query-value mechanic by just dropping it while making the entire context the latent space. That will outperform attention, nearly guaranteed. It'll also make even Google's cluster networks meltdown. Attention is one of those innovations that came mostly from realizing you had better hardware than everybody else and asking yourself how to use it. It's still quite the accomplishment, they had to get it working. But nobody else was really capable of making this leap.
Re: A walk through of the DeltaNet family of linear attention variants
#14I could never get this about modern machine/deep learning or even the Transformers. Yes, it's not exactly rocket science, but when I see the data flow diagrams, it's not clear what is calculated in real time or multiple steps. Is it really one big computation f(g(h(x)))?
It's all vibes.
Re: A walk through of the DeltaNet family of linear attention variants
#15I could never get this about modern machine/deep learning or even the Transformers. Yes, it's not exactly rocket science, but when I see the data flow diagrams, it's not clear what is calculated in real time or multiple steps. Is it really one big computation f(g(h(x)))?
[deleted]
Re: A walk through of the DeltaNet family of linear attention variants
#16No, you couldn't have. There are plenty of ML innovations that when push comes to shove only depend on having access to more compute, but this is one of the worst examples I've ever seen. I always thought that the jump from LSTM/GRU -> Attention wasn't a particularly big one. Instead of partial unroll, do a full unroll. Why not (because it's too expensive, that's why not). Every component was known, and everybody any…
I agree 100%. This field is not amenable to progress from people with a pen sitting in a corner proving theorems. The math is mostly uncertain vibes and to test it you need millions of dollars of compute. Smart loners just can't.
Re: A walk through of the DeltaNet family of linear attention variants
#17Relevant XKCD
Re: A walk through of the DeltaNet family of linear attention variants
#18Loved this incremental evolution, things gets way more understandable...usually xD
Re: A walk through of the DeltaNet family of linear attention variants
#19Machine learning could need, and probably has needed, some unified math notation for the past 15 years IMO. With that said, it was worse back in the day - when ML papers were the products of researchers from all over, you'd see some wild notation.
Many will likely disagree with me, but inconsistent notation (across papers!) is to me friction. At least in this article the author explicitly explains the notation at the very start...that is not always the case. Rarely, even.
EDIT: Didn't even notice the notation switch, much appreciated.
Re: A walk through of the DeltaNet family of linear attention variants
#20I could never get this about modern machine/deep learning or even the Transformers. Yes, it's not exactly rocket science, but when I see the data flow diagrams, it's not clear what is calculated in real time or multiple steps. Is it really one big computation f(g(h(x)))?
Yes.
Each token prediction is one big function call. Then you just recursively generate more tokens until run out of context or the model predicts a next token indicating end of sequence. Technically the model outputs a matrix where the last row is a probability distribution, but I’m counting sampling from it as part of the chain. Hundreds of billions of dollars has gone into just making the function fatter and gradually changing pieces here and there.