Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A walk through of the DeltaNet family of linear attention variants

blog.doubleword.ai

Post reply on HN
Print view View original

121–130 of 134 posts

Re: A walk through of the DeltaNet family of linear attention variants

#121

Post by xp84 »

abixb wrote: post #101
phoghed wrote: post #86

Earlier quoted context omitted.

Not sure I follow, what are you doing with these philosopher kings after you mint them?

Well, the goal was to make them come up with cutting-edge theories and/or hypothesis to the most pressing problems facing humanity. Kids will naturally specialize (just like... MoE models?) as their find their groove while growing up, but the goal is to make sure highly gifted kids from diverse background get highest quality space to think without being moulded or "adulterated" by the noise of pop culture and "everyd…

> make sure highly gifted kids from diverse background

It would be ideal to get those gifted kids before they spend 5-6 years in the 'real world' with all the "Baby Shark," Roblox, and various other brainrot, but if you pick kids randomly from a very early age, most of them will probably not be all that gifted. Though I suppose maybe you could rely on genetic heritage (parents of very high IQ) and/or just do a large number of them and just assume you'll luck into enough geniuses to be worth your while.

Re: A walk through of the DeltaNet family of linear attention variants

#124

Post by olalonde »

glaslong wrote: post #90
olalonde wrote: post #58

Earlier quoted context omitted.

I never understood people who preferred traditional math notation (e.g. single letter symbols, weird characters like ∣q⟩ instead of writing down an explicit type, etc.). I guess the main advantage is terseness? To me, the mathematical expressions would be so much easier to understand if they were just written in pseudo code or an actual programming language like Python.

You really just get tired of writing/reading "AbstractJavaSerializerBeanFactoryFactoryAbstactMutatorFactoryAccessEnterpriseBeanFactory()" over and over again. So you and all your peers agree to call that procedure "ẽ"

Mathematicians probably spend less time writing than programmers.

Re: A walk through of the DeltaNet family of linear attention variants

#125

Post by krackers »

There was a longform post on twitter which went through the same derivation at a bit higher level

https://x.com/waterloo_intern/article/2081762065392541951

and in particular this image which clarifies the key essential difference between liner attention and delta network by examining the case of two tokens with same key but different value

https://pbs.twimg.com/media/HOPCc7BaEAAQDtO.jpg?format=jpg&n...

I think for comparison it would also have been good to have how original quadratic attention handles it: since both keys are identical, the attention would be "evenly divided" between both values so the final output would be the average of both values, as opposed to the latest value

Re: A walk through of the DeltaNet family of linear attention variants

#126

Post by krackers »

joe_the_user wrote: post #95

Overall, all the different linear attentions out there are approximations the original (quadratic) attention and this is important for the whole "AI" enterprise[2]. Original attention involves (very crudely) an approach of scanning how every token (roughly a word) relates every other token and training a classic neural network on related tokens - to get either language translation or next word prediction (and next wo…

I had hoped the OP article would have gone into more depth on the intuition as to why you'd expect deltanets to work at all. It seems like we're going back to LSTMs and RNNs where you compress the history back into a fixed-size hidden state. Aside from the easy parallelization in training, I thought that Attention worked much better because it got over this fundamental bottleneck and just let every token interact with every past token (of course you pay for it in compute, but ultimately that's what got us a GPT).

I guess that's probably why you still need some MLA layers in there.

Re: A walk through of the DeltaNet family of linear attention variants

#127

Post by aeternum »

crubier wrote: post #112
kurthr wrote: post #79

Earlier quoted context omitted.

Longer to write and generally a numerical rather than symbolic representation.

Longer to write: Don't care, that's kinda the point. I'd rather read a descriptive name in english than a greek letter with a tilda and a dot subscript. Generally a numerical rather than symbolic representation: Meh https://docs.sympy.org/latest/modules/functions/elementary.h...

At least one benefit is you can now leverage LLMs to basically translate entire math textbooks into your preferred notation.

I would be curious whether you find it easier. I kinda think it'll be too many parenthesis.

Re: A walk through of the DeltaNet family of linear attention variants

#128

Post by TeMPOraL »

glaslong wrote: post #90
olalonde wrote: post #58

Earlier quoted context omitted.

I never understood people who preferred traditional math notation (e.g. single letter symbols, weird characters like ∣q⟩ instead of writing down an explicit type, etc.). I guess the main advantage is terseness? To me, the mathematical expressions would be so much easier to understand if they were just written in pseudo code or an actual programming language like Python.

You really just get tired of writing/reading "AbstractJavaSerializerBeanFactoryFactoryAbstactMutatorFactoryAccessEnterpriseBeanFactory()" over and over again. So you and all your peers agree to call that procedure "ẽ"

Yes. Specifically, that thing on top is the "AbstractJAvaSerializerFactoryFactory over AbstractMutatorFactoryAccess over Factory; "e" is just EnterpriseBean.

Re: A walk through of the DeltaNet family of linear attention variants

#129

Post by spwa4 »

p1esk wrote: post #41
spwa4 wrote: post #13

No, you couldn't have. There are plenty of ML innovations that when push comes to shove only depend on having access to more compute, but this is one of the worst examples I've ever seen. I always thought that the jump from LSTM/GRU -> Attention wasn't a particularly big one. Instead of partial unroll, do a full unroll. Why not (because it's too expensive, that's why not). Every component was known, and everybody any…

replace they key-query-value mechanic by just dropping it while making the entire context the latent space. What do you mean by this? Like concatenating all token embeddings into one large vector?

Yes. Just have the entire context visible to everything, all the time.

It's just one way to increase the expressiveness of the whole network. In general one could say that coming up with ways to greatly increase the expressiveness of neural nets is not hard. Or it's hard because it blows up compute. Meanwhile the human mind demonstrates that you can drop compute by 5-6 orders of magnitude without losing expressiveness to save power.

Re: A walk through of the DeltaNet family of linear attention variants

#130

Post by alf42red »

I've only read 1-4 without prior knowledge of qkv attention and I like the explanation. The only thing I didn't get from the text was why we divide by the square root of d_k, but ChatGPT explained it was to counter variance scaling with the dimension of the matrix. I probably couldn't have come up with this myself, but I feel that it actually makes sense now and I like that deltanet attention "learns a learning rate" according to one interpretation of beta from the text, if I'm understanding it correctly. Great article!
Post reply on HN