Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A walk through of the DeltaNet family of linear attention variants

blog.doubleword.ai

Post reply on HN
Print view View original

41–50 of 134 posts

Re: A walk through of the DeltaNet family of linear attention variants

#41

Post by p1esk »

spwa4 wrote: post #13

No, you couldn't have. There are plenty of ML innovations that when push comes to shove only depend on having access to more compute, but this is one of the worst examples I've ever seen. I always thought that the jump from LSTM/GRU -> Attention wasn't a particularly big one. Instead of partial unroll, do a full unroll. Why not (because it's too expensive, that's why not). Every component was known, and everybody any…

replace they key-query-value mechanic by just dropping it while making the entire context the latent space.

What do you mean by this? Like concatenating all token embeddings into one large vector?

Re: A walk through of the DeltaNet family of linear attention variants

#43

Post by stuxnet79 »

bee_rider wrote: post #35

Where do linear algebra folks go to get started with ML stuff? It seems pretty easy but the hardware is expensive.

> It seems pretty easy but the hardware is expensive.

Huh?

If your aim is to truly 'get started' with ML then hardware is absolutely not a bottleneck (either local or cloud).

Remember that ML is much more than LLMs. Even modern day LLMs can be quantized to a point where they can run on local hardware although their capabilities won't be as impressive.

I would recommend looking into some of Andrej Karpathy's videos if you want a grasp of the basics.

Re: A walk through of the DeltaNet family of linear attention variants

#44

Post by sva_ »

bee_rider wrote: post #35

Where do linear algebra folks go to get started with ML stuff? It seems pretty easy but the hardware is expensive.

I think Karpathys nn zero to hero is a good starting point. And you can experiment on small networks using pretty normal hardware.

Re: A walk through of the DeltaNet family of linear attention variants

#46

Post by baq »

londons_explore wrote: post #45
rekshaw wrote: post #2

after a cursory read, I can confidently say I could not, in fact, have come up with Kimi Delta Attention.

The notation looks complex, but underneath it's all just adding and multiplying. Nothing complex

as is practically all of transformer maths if you squint hard enough...

Re: A walk through of the DeltaNet family of linear attention variants

#47

Post by ReactiveJelly »

londons_explore wrote: post #45
rekshaw wrote: post #2

after a cursory read, I can confidently say I could not, in fact, have come up with Kimi Delta Attention.

The notation looks complex, but underneath it's all just adding and multiplying. Nothing complex

"I invented a new algorithm"

"New algorithm, or fmadd?"

"... fmadd."

Re: A walk through of the DeltaNet family of linear attention variants

#48

Post by penguin_booze »

rekshaw wrote: post #2

after a cursory read, I can confidently say I could not, in fact, have come up with Kimi Delta Attention.

"you could have..." is among the top insulting phrases used by maths-adjacent people. Others in that league are "it should now be obvious...", "it's abundantly clear...", "it can be easily shown that...", "this is nothing but..." etc.

The rest of us reading this are like, holy batman, what the fuck was that?!

Re: A walk through of the DeltaNet family of linear attention variants

#49

Post by teach »

londons_explore wrote: post #45
rekshaw wrote: post #2

after a cursory read, I can confidently say I could not, in fact, have come up with Kimi Delta Attention.

The notation looks complex, but underneath it's all just adding and multiplying. Nothing complex

Grand Theft Auto VI looks complex, but underneath it's all just ones and zeros and NAND

Re: A walk through of the DeltaNet family of linear attention variants

#50

Post by Razengan »

penguin_booze wrote: post #48
rekshaw wrote: post #2

after a cursory read, I can confidently say I could not, in fact, have come up with Kimi Delta Attention.

"you could have..." is among the top insulting phrases used by maths-adjacent people. Others in that league are "it should now be obvious...", "it's abundantly clear...", "it can be easily shown that...", "this is nothing but..." etc. The rest of us reading this are like, holy batman, what the fuck was that?!

Right next to "Learn More" by software UI designers.
Post reply on HN