Side note, before you ask: yes, bra-ket notation is called like that because of the brackets.
A walk through of the DeltaNet family of linear attention variants
31–40 of 134 posts
Re: A walk through of the DeltaNet family of linear attention variants
#32>You Could Have Come Up With Kimi Delta Attention
What? Little old me! Well, then, let's have a look...
> (First paragraph)
> A note on notation: this article defaults to bra-ket notation because (in my quantum-inspired opinion) it makes the shapes in this derivation very clear. The Math notation switch above rewrites every equation using conventional bold vectors and explicit transposes instead. In bra-ket mode, ∣ q ⟩ ∣q⟩ is a column vector, ⟨ k ∣ ⟨k∣ is a row vector, ⟨ k ∣ q ⟩ ⟨k∣q⟩ is a number, and ∣ v ⟩ ⟨ k ∣ ∣v⟩⟨k∣ is a matrix. Vectors face right by default, while keys face left when written into the linear-attention state. We work with one causal attention head and real-valued vectors, assume DeltaNet’s keys are normalized, and let the state map from key space to value space.
Hmm... Guess not!
Re: A walk through of the DeltaNet family of linear attention variants
#33>You Could Have Come Up With Kimi Delta Attention What? Little old me! Well, then, let's have a look... > (First paragraph) > A note on notation: this article defaults to bra-ket notation because (in my quantum-inspired opinion) it makes the shapes in this derivation very clear. The Math notation switch above rewrites every equation using conventional bold vectors and explicit transposes instead. In bra-ket mode, ∣ q…
I love that they let you switch to a more common q'k notation!
Re: A walk through of the DeltaNet family of linear attention variants
#34LLM written for sure: > The identity [...] is the whole trick. The outer product is a matrix; the inner product is a number. We no longer store every past key and value. We store their summed outer products in the fixed-size state S_t.
This is what you get when you prompt claude to avoid –
Re: A walk through of the DeltaNet family of linear attention variants
#35Where do linear algebra folks go to get started with ML stuff? It seems pretty easy but the hardware is expensive.
Re: A walk through of the DeltaNet family of linear attention variants
#36Re: A walk through of the DeltaNet family of linear attention variants
#37after a cursory read, I can confidently say I could not, in fact, have come up with Kimi Delta Attention.
They lost me just describing the notation lol
Re: A walk through of the DeltaNet family of linear attention variants
#38You know its a doozy when the author writes a disclaimer at the top saying that bra-ket notation was chosen in order to make the algorithm and data structures clearer.
One of the more annoying parts of my physics study was getting used to the new matrix multiplication notation they came up with every semester.
bra-ket is the (most?) general form of tensor manipulation.
Raising and lowering operators for summation notation are the beginner tools for covariant derivatives of the metric tensor.
Christoffel symbols are where it's at, if you need to write out the Ricci tensor. The more constrained the space the more concise the notation can be.
Note that MechE tensor notation has an even more compact (eigen) form for principal stresses.
Re: A walk through of the DeltaNet family of linear attention variants
#39Hmm. Hmmm. Hmm. HMMM. Hmm.
Yep! I know some of these words.
Re: A walk through of the DeltaNet family of linear attention variants
#40A visualized tutorial: https://snowchord.com/blog/linear-attention-visualized/