A walk through of the DeltaNet family of linear attention variants • Hacker News Board

Re: A walk through of the DeltaNet family of linear attention variants

#81

Post by abixb » Tue, Jul 28, 2026, 6:38 PM UTC

xp84 wrote: post #66

When I see these types of articles and headlines, it just makes me supremely grateful for all the many people far smarter[1] than me. And humbles me, too, since I actually passed for a "very smart person" in places like high school and undergrad. In fact, I'm 'smart' for an average person, but there are definitely millions of people who make me look like a rube in comparison. [1] I specifically mean those who are abl…

Yes. I continue to believe that humans will still be the source of the vast majority of novel ideas, even as they increasingly use AI-related tools to accelerate their works.

One of the though experiments I ran with one of my friends during a recent conversation over drinks was this: raising a bunch of "control group" kids away from the screens and the algorithmic ocean of "normie-tier content," and in a very learner-friendly setting with hyper-strict control on the quality of media and source material they get access to, just like we've been doing it with frontier models. Think of it like a monastery but for kids, while teaching them all the latest advances in our understanding of reality through mathematics, engineering, computer science, deep learning, and whatnot.

What I'm getting at it is that we might still need super smart people to push the boundaries of knowledge while using super-advanced AI tools, and anyone who says AI will "completely replace" humans are just misguided. We will always need super smart people with largely unadulterated thinking.

Re: A walk through of the DeltaNet family of linear attention variants

#82

Post by wrs » Tue, Jul 28, 2026, 6:51 PM UTC

jameshart wrote: post #70

penguin_booze wrote: post #48

Earlier quoted context omitted.

"you could have..." is among the top insulting phrases used by maths-adjacent people. Others in that league are "it should now be obvious...", "it's abundantly clear...", "it can be easily shown that...", "this is nothing but..." etc. The rest of us reading this are like, holy batman, what the fuck was that?!

Math educators like Grant Sanderson (3blue1brown) use it in a very specific way: the goal of a mathematical explanation is to make the learner feel like they could have come up with something. And a really good mathematical communicator can absolutely do that. A piece like this which uses it in a headline but in no way makes an average reader feel like they could have come up with it is just badly misjudging how good…

I don’t think “you” in these titles ever really refers to an “average reader”. Some familiarity with the field is required. Imagine how non-programmers (and many programmers) feel about some examples I just Googled:

“You Could Have Invented Parser Combinators”

“You Could Have Invented Container Runtimes”

“You Could Have Invented Git”

Given the references to “mathematicians”, I think this reaction is more about an unfamiliarity with the concept of applied mathematics, which is ironic for practitioners in a field containing so much that is (or should be) regarded that way.

Software used to be all about “discrete math”, logic, maybe a bit of queuing theory…and now suddenly linear algebra and statistics became important. Don’t panic, it’s just another textbook on the shelf.

Re: A walk through of the DeltaNet family of linear attention variants

#83

Post by luciana1u » Tue, Jul 28, 2026, 6:51 PM UTC

love the toggle between math notation and physics notation. two flavors of confusion, nicely packaged.

Re: A walk through of the DeltaNet family of linear attention variants

#84

Post by tyre » Tue, Jul 28, 2026, 6:55 PM UTC

dfee wrote: post #72

htrp wrote: post #60

Earlier quoted context omitted.

math notation doesn't bias towards English language understanding like pseudocode

but it biases towards the latin alphabet. so your point is diminished.

and Greek!

Re: A walk through of the DeltaNet family of linear attention variants

#85

Post by Barbing » Tue, Jul 28, 2026, 6:55 PM UTC

devy wrote: post #75

rekshaw wrote: post #2

after a cursory read, I can confidently say I could not, in fact, have come up with Kimi Delta Attention.

Doubleword AI is conducting a classic textbook marketing trick called newsjacking. Writing a detailed technical post behind the news of Kimi K3 and KDA algorithm with an audacious title like "You Could Have Invent Breakthrough It too" they are pre-filtering out the ones who couldn't comprehend with quick read (myself included) and attracting the ones who agreed with the blog post. At the end with a strong CTA to prom…

“Kimi Delta Attention” because “Kimi K3 Delta Attention (oh that’s just our little internal name for it as a joke)” passes no sniff tests.

Re: A walk through of the DeltaNet family of linear attention variants

#86

Post by phoghed » Tue, Jul 28, 2026, 6:55 PM UTC

abixb wrote: post #81

xp84 wrote: post #66

When I see these types of articles and headlines, it just makes me supremely grateful for all the many people far smarter[1] than me. And humbles me, too, since I actually passed for a "very smart person" in places like high school and undergrad. In fact, I'm 'smart' for an average person, but there are definitely millions of people who make me look like a rube in comparison. [1] I specifically mean those who are abl…

Yes. I continue to believe that humans will still be the source of the vast majority of novel ideas, even as they increasingly use AI-related tools to accelerate their works. One of the though experiments I ran with one of my friends during a recent conversation over drinks was this: raising a bunch of "control group" kids away from the screens and the algorithmic ocean of "normie-tier content," and in a very learner…

Not sure I follow, what are you doing with these philosopher kings after you mint them?

Re: A walk through of the DeltaNet family of linear attention variants

#87

Post by OkayPhysicist » Tue, Jul 28, 2026, 6:55 PM UTC

olalonde wrote: post #58

TrackerFF wrote: post #19

Machine learning could need, and probably has needed, some unified math notation for the past 15 years IMO. With that said, it was worse back in the day - when ML papers were the products of researchers from all over, you'd see some wild notation. Many will likely disagree with me, but inconsistent notation (across papers!) is to me friction. At least in this article the author explicitly explains the notation at the…

I never understood people who preferred traditional math notation (e.g. single letter symbols, weird characters like ∣q⟩ instead of writing down an explicit type, etc.). I guess the main advantage is terseness? To me, the mathematical expressions would be so much easier to understand if they were just written in pseudo code or an actual programming language like Python.

Terseness is a significant advantage in pattern recognition. If you write a long, detailed breakdown of every step, not only are you spending a bunch of time writing, you're also obscuring the natural symmetries of the statement.

It's like saying "I never understood people who prefer to use functions instead of inlining everything". Adding a bunch of visual noise to a statement doesn't improve comprehension.

Re: A walk through of the DeltaNet family of linear attention variants

#88

Post by glaslong » Tue, Jul 28, 2026, 6:56 PM UTC

rekshaw wrote: post #2

after a cursory read, I can confidently say I could not, in fact, have come up with Kimi Delta Attention.

* a completely different "you" who spent countless hours gaining expertise on a wholly diverged life path

Re: A walk through of the DeltaNet family of linear attention variants

#89

Post by whatsakandr » Tue, Jul 28, 2026, 6:57 PM UTC

TrackerFF wrote: post #19

Machine learning could need, and probably has needed, some unified math notation for the past 15 years IMO. With that said, it was worse back in the day - when ML papers were the products of researchers from all over, you'd see some wild notation. Many will likely disagree with me, but inconsistent notation (across papers!) is to me friction. At least in this article the author explicitly explains the notation at the…

I used to think this, then I realized that the amount of time you spend with equations is so much more than code, and the terseness makes them much easier to read once you know what the symbols are.

Also, letters avoid having to name them, naming being a hard problem and all.

Re: A walk through of the DeltaNet family of linear attention variants

#90

Post by glaslong » Tue, Jul 28, 2026, 7:01 PM UTC

olalonde wrote: post #58

TrackerFF wrote: post #19

Machine learning could need, and probably has needed, some unified math notation for the past 15 years IMO. With that said, it was worse back in the day - when ML papers were the products of researchers from all over, you'd see some wild notation. Many will likely disagree with me, but inconsistent notation (across papers!) is to me friction. At least in this article the author explicitly explains the notation at the…

I never understood people who preferred traditional math notation (e.g. single letter symbols, weird characters like ∣q⟩ instead of writing down an explicit type, etc.). I guess the main advantage is terseness? To me, the mathematical expressions would be so much easier to understand if they were just written in pseudo code or an actual programming language like Python.

You really just get tired of writing/reading "AbstractJavaSerializerBeanFactoryFactoryAbstactMutatorFactoryAccessEnterpriseBeanFactory()" over and over again.

So you and all your peers agree to call that procedure "ẽ"