why are you using braket notation?
He has a master's degree in physics from Oxford. Also there is a toggle to normal notation. Well, CS notation. I'm not a fan of transpose marks everywhere. I like an even more mathematics notation.
A walk through of the DeltaNet family of linear attention variants
21–30 of 134 posts
Re: A walk through of the DeltaNet family of linear attention variants
#22Re: A walk through of the DeltaNet family of linear attention variants
#23At first I felt bad about not having come up with this solution. But then I realized I have problems with writing binary search by myself in JS and immediately felt better.
Now way I could have come up with Kimi Delta Attention.
Re: A walk through of the DeltaNet family of linear attention variants
#24I could never get this about modern machine/deep learning or even the Transformers. Yes, it's not exactly rocket science, but when I see the data flow diagrams, it's not clear what is calculated in real time or multiple steps. Is it really one big computation f(g(h(x)))?
What's your distinction between real time vs multiple steps? All computation is done in steps.
Is it all one big computation? Its turtles all the way down.
Re: A walk through of the DeltaNet family of linear attention variants
#25LLM written for sure:
> The identity [...] is the whole trick. The outer product is a matrix; the inner product is a number. We no longer store every past key and value. We store their summed outer products in the fixed-size state S_t.
Re: A walk through of the DeltaNet family of linear attention variants
#26Re: A walk through of the DeltaNet family of linear attention variants
#27Re: A walk through of the DeltaNet family of linear attention variants
#28LLM written for sure: > The identity [...] is the whole trick. The outer product is a matrix; the inner product is a number. We no longer store every past key and value. We store their summed outer products in the fixed-size state S_t.
Ya, probably started with asking for a buzzy title.
Re: A walk through of the DeltaNet family of linear attention variants
#29At first I felt bad about not having come up with this solution. But then I realized I have problems with writing binary search by myself in JS and immediately felt better. Now way I could have come up with Kimi Delta Attention.
Lots of linear algebra codes are actually “easy to write” in a way. It isn’t like conventional CS where you are always going a bunch of recursive nonsense going on. There should be mathematical relationships between all of the variables, there are well implemented libraries for the common mathematical concepts, and it is rare to need to go more than a couple loops deep (anything more complex than that should get shunted off into a library anyway).
Re: A walk through of the DeltaNet family of linear attention variants
#30after a cursory read, I can confidently say I could not, in fact, have come up with Kimi Delta Attention.
Yeah, pretty sure the “you” in “you could have” is a different “you” than “we”.