A walk through of the DeltaNet family of linear attention variants • Hacker News Board

Re: A walk through of the DeltaNet family of linear attention variants

#71

Post by crubier » Tue, Jul 28, 2026, 6:27 PM UTC

olalonde wrote: post #58

TrackerFF wrote: post #19

Machine learning could need, and probably has needed, some unified math notation for the past 15 years IMO. With that said, it was worse back in the day - when ML papers were the products of researchers from all over, you'd see some wild notation. Many will likely disagree with me, but inconsistent notation (across papers!) is to me friction. At least in this article the author explicitly explains the notation at the…

I never understood people who preferred traditional math notation (e.g. single letter symbols, weird characters like ∣q⟩ instead of writing down an explicit type, etc.). I guess the main advantage is terseness? To me, the mathematical expressions would be so much easier to understand if they were just written in pseudo code or an actual programming language like Python.

This 1,000%

Trying to read any math paper is basically like trying to read CodeGolf.

Re: A walk through of the DeltaNet family of linear attention variants

#72

Post by dfee » Tue, Jul 28, 2026, 6:27 PM UTC

htrp wrote: post #60

olalonde wrote: post #58

Earlier quoted context omitted.

I never understood people who preferred traditional math notation (e.g. single letter symbols, weird characters like ∣q⟩ instead of writing down an explicit type, etc.). I guess the main advantage is terseness? To me, the mathematical expressions would be so much easier to understand if they were just written in pseudo code or an actual programming language like Python.

math notation doesn't bias towards English language understanding like pseudocode

but it biases towards the latin alphabet. so your point is diminished.

Re: A walk through of the DeltaNet family of linear attention variants

#73

Post by crubier » Tue, Jul 28, 2026, 6:27 PM UTC

aeternum wrote: post #65

olalonde wrote: post #58

Earlier quoted context omitted.

I never understood people who preferred traditional math notation (e.g. single letter symbols, weird characters like ∣q⟩ instead of writing down an explicit type, etc.). I guess the main advantage is terseness? To me, the mathematical expressions would be so much easier to understand if they were just written in pseudo code or an actual programming language like Python.

Math notation ultimately is pseudo code just with mostly single letter variables and many operators that are encoded purely by position thus not even requiring a symbol. Remember that the oft-used e^x is actually an infinite series, even writing it out in summation form would be quite verbose given its frequency in many equations.

exp(x)

Re: A walk through of the DeltaNet family of linear attention variants

#74

Post by nurettin » Tue, Jul 28, 2026, 6:28 PM UTC

It is heartwarming to see how sarcasm turns into a celebration of mediocrity.

Re: A walk through of the DeltaNet family of linear attention variants

#75

Post by devy » Tue, Jul 28, 2026, 6:30 PM UTC

rekshaw wrote: post #2

after a cursory read, I can confidently say I could not, in fact, have come up with Kimi Delta Attention.

Doubleword AI is conducting a classic textbook marketing trick called newsjacking.

Writing a detailed technical post behind the news of Kimi K3 and KDA algorithm with an audacious title like "You Could Have Invent Breakthrough It too" they are pre-filtering out the ones who couldn't comprehend with quick read (myself included) and attracting the ones who agreed with the blog post. At the end with a strong CTA to promoting their 10x cheapter open weight model AI inference and hiring too.

Good job Doubleword, I see what you are doing there.

Re: A walk through of the DeltaNet family of linear attention variants

#76

Post by dboreham » Tue, Jul 28, 2026, 6:33 PM UTC

olalonde wrote: post #58

TrackerFF wrote: post #19

Machine learning could need, and probably has needed, some unified math notation for the past 15 years IMO. With that said, it was worse back in the day - when ML papers were the products of researchers from all over, you'd see some wild notation. Many will likely disagree with me, but inconsistent notation (across papers!) is to me friction. At least in this article the author explicitly explains the notation at the…

I never understood people who preferred traditional math notation (e.g. single letter symbols, weird characters like ∣q⟩ instead of writing down an explicit type, etc.). I guess the main advantage is terseness? To me, the mathematical expressions would be so much easier to understand if they were just written in pseudo code or an actual programming language like Python.

Well, humanity struggled for 2000 years trying to do mathematics without notation, so its benefit is not to be sniffed at. But really it's just an APL-vs-Fortran type debate. They're not fundamentally different. Remember also that Ramanujan had to re-use paper it was so costly/hard to find.

Re: A walk through of the DeltaNet family of linear attention variants

#77

Post by SkyBelow » Tue, Jul 28, 2026, 6:34 PM UTC

aeternum wrote: post #65

olalonde wrote: post #58

Earlier quoted context omitted.

I never understood people who preferred traditional math notation (e.g. single letter symbols, weird characters like ∣q⟩ instead of writing down an explicit type, etc.). I guess the main advantage is terseness? To me, the mathematical expressions would be so much easier to understand if they were just written in pseudo code or an actual programming language like Python.

Math notation ultimately is pseudo code just with mostly single letter variables and many operators that are encoded purely by position thus not even requiring a symbol. Remember that the oft-used e^x is actually an infinite series, even writing it out in summation form would be quite verbose given its frequency in many equations.

I feel the magic in math is that the notation is to a more universal sort of language, and the e^x captures that. e^x means that infinite series, but it means a few other things at the same time. That those different things are sometimes related, that e^2 means e*e, a simple enough operation most can understand, but that it also means that infinite series with 2 in place of x, is part of the complexity of math that makes it quite different than a classical programming language. With most programming languages, someone assigned those equivalent meaning, and while it might be important to understand why, it shows you the mind of the who designed the language. But, in math (and in true Computer Science), when you see an equivalence of this sort, whose mind are you now glimpsing into?

But, for someone just starting out (or even an expert who just wandered into a new area), this becomes a barrier to understanding. Sometimes we need the mathematical summary, what tells us the immediately answer we are interested in, perhaps even if it loses the deeper nuances that are beneficial for the experts more fluent in the language.

(And yes, all math notation is human created, so it makes my point quite a bit messier in practice.)

Re: A walk through of the DeltaNet family of linear attention variants

#78

Post by cubefox » Tue, Jul 28, 2026, 6:34 PM UTC

ozgung wrote: post #54

penguin_booze wrote: post #48

Earlier quoted context omitted.

"you could have..." is among the top insulting phrases used by maths-adjacent people. Others in that league are "it should now be obvious...", "it's abundantly clear...", "it can be easily shown that...", "this is nothing but..." etc. The rest of us reading this are like, holy batman, what the fuck was that?!

Also the proof is so trivial that it’s left to the reader.

In the future this might be replaced with "you can verify this fact by asking an LLM of your choice". Similar to how people in chat arguments already post screenshots of an LLM answer to show that their opinion is correct.

Re: A walk through of the DeltaNet family of linear attention variants

#79

Post by kurthr » Tue, Jul 28, 2026, 6:35 PM UTC

crubier wrote: post #73

aeternum wrote: post #65

Earlier quoted context omitted.

Math notation ultimately is pseudo code just with mostly single letter variables and many operators that are encoded purely by position thus not even requiring a symbol. Remember that the oft-used e^x is actually an infinite series, even writing it out in summation form would be quite verbose given its frequency in many equations.

exp(x)

Longer to write and generally a numerical rather than symbolic representation.

Re: A walk through of the DeltaNet family of linear attention variants

#80

Post by sodapopcan » Tue, Jul 28, 2026, 6:36 PM UTC

Ohhhhh Diag(αt), right. I was almost there but had left the placeholder "Diag(foo)" and never noticed. I now see is why I didn't come up with it first. So close!