Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A walk through of the DeltaNet family of linear attention variants

blog.doubleword.ai

Post reply on HN
Print view View original

111–120 of 134 posts

Re: A walk through of the DeltaNet family of linear attention variants

#111

Post by arkj »

mezark wrote: post #96
devy wrote: post #75

Earlier quoted context omitted.

Doubleword AI is conducting a classic textbook marketing trick called newsjacking. Writing a detailed technical post behind the news of Kimi K3 and KDA algorithm with an audacious title like "You Could Have Invent Breakthrough It too" they are pre-filtering out the ones who couldn't comprehend with quick read (myself included) and attracting the ones who agreed with the blog post. At the end with a strong CTA to prom…

lol - co-founder of Doubleword here. honoured you think we have sophisticated enough marketing to 'newsjack'. What actually happened is my cofounder wrote it over the weekend because he's a mega-nerd and put it live yesterday. I hadn't even read it until I saw this hacker news thread lol We're just a group of guys and gals who like inference!

The blogs from doubleword are indeed full precision. A great team doing a fantastic job.

Re: A walk through of the DeltaNet family of linear attention variants

#112

Post by crubier »

kurthr wrote: post #79
crubier wrote: post #73

Earlier quoted context omitted.

exp(x)

Longer to write and generally a numerical rather than symbolic representation.

Longer to write: Don't care, that's kinda the point. I'd rather read a descriptive name in english than a greek letter with a tilda and a dot subscript.

Generally a numerical rather than symbolic representation: Meh https://docs.sympy.org/latest/modules/functions/elementary.h...

Re: A walk through of the DeltaNet family of linear attention variants

#113

Post by dmd »

abixb wrote: post #104
dmd wrote: post #91

Earlier quoted context omitted.

You should read 'Anathem' by Neal Stephenson, which goes into great deal about this kind of establishment.

Thank you for the recommendation. I'm yet to checkout any of Neal Stephenson's works, which is real unfortunate.

I'd say Anathem is the best of the bunch, and has held up best over time.

Re: A walk through of the DeltaNet family of linear attention variants

#114

Post by moffkalast »

cubefox wrote: post #78
ozgung wrote: post #54

Earlier quoted context omitted.

Also the proof is so trivial that it’s left to the reader.

In the future this might be replaced with "you can verify this fact by asking an LLM of your choice". Similar to how people in chat arguments already post screenshots of an LLM answer to show that their opinion is correct.

Counterexample left as an exercise for Fable

Re: A walk through of the DeltaNet family of linear attention variants

#115

Post by aesthesia »

wrs wrote: post #82
jameshart wrote: post #70

Earlier quoted context omitted.

Math educators like Grant Sanderson (3blue1brown) use it in a very specific way: the goal of a mathematical explanation is to make the learner feel like they could have come up with something. And a really good mathematical communicator can absolutely do that. A piece like this which uses it in a headline but in no way makes an average reader feel like they could have come up with it is just badly misjudging how good…

I don’t think “you” in these titles ever really refers to an “average reader”. Some familiarity with the field is required. Imagine how non-programmers (and many programmers) feel about some examples I just Googled: “You Could Have Invented Parser Combinators” “You Could Have Invented Container Runtimes” “You Could Have Invented Git” Given the references to “mathematicians”, I think this reaction is more about an unf…

I think the origin of the title formula is Timothy Chow's "You Could Have Invented Spectral Sequences" (https://www.ams.org/notices/200601/fea-chow.pdf), which is certainly not aimed at an average reader.

Re: A walk through of the DeltaNet family of linear attention variants

#116

Post by malwrar »

brcmthrowaway wrote: post #103
malwrar wrote: post #20

Earlier quoted context omitted.

Yes. Each token prediction is one big function call. Then you just recursively generate more tokens until run out of context or the model predicts a next token indicating end of sequence. Technically the model outputs a matrix where the last row is a probability distribution, but I’m counting sampling from it as part of the chain. Hundreds of billions of dollars has gone into just making the function fatter and gradu…

I remember the concept of layers, as essentially defining the matrix math dimensions. And for a given model/framework, they were static. That always bugged me (not very dynamic).. is this still the case?

The sizes for the matrices are decided up front (“hyperparameters”, more or less guesstimated) and the values that are learned (“parameters”, for completeness) and remain static after training. The layering just refers to the core “transformer block” being chained inside the model that contains most of the weights. Those themselves are pretty simple, usually just an MLP followed by an attention function, with variations on formulation & “multi-heading” in most of the new models. Behind the terms and math and inner architectural choices, it’s pretty much still the same GPT pattern of ‘“lie” = decode(sample(lm_head(T_1(T_2(…T_n(encode(“the cake is a”))))))’, with T_i(…) being the ith layer.

You know, I haven’t kept up with MoE and etc where there’s a bit of selection going on, so I should probably be more humble. I think new work has only added different “paths down the same hill” though (no recurrence, just select different matrices), but could be wrong there. I don’t think I’m wrong on my general intuition, just want to be epistemically honest!

Re: A walk through of the DeltaNet family of linear attention variants

#117

Post by Barbing »

Barbing wrote: post #85
devy wrote: post #75

Earlier quoted context omitted.

Doubleword AI is conducting a classic textbook marketing trick called newsjacking. Writing a detailed technical post behind the news of Kimi K3 and KDA algorithm with an audacious title like "You Could Have Invent Breakthrough It too" they are pre-filtering out the ones who couldn't comprehend with quick read (myself included) and attracting the ones who agreed with the blog post. At the end with a strong CTA to prom…

“Kimi Delta Attention” because “Kimi K3 Delta Attention (oh that’s just our little internal name for it as a joke)” passes no sniff tests.

(HN titled updated from previous “You Could Have Come Up With Kimi Delta Attention” … gotta check if someone maintains an “Editing The Gray Lady” for HN, for title histories and derankings)

Re: A walk through of the DeltaNet family of linear attention variants

#118

Post by smallmancontrov »

OkayPhysicist wrote: post #87
olalonde wrote: post #58

Earlier quoted context omitted.

I never understood people who preferred traditional math notation (e.g. single letter symbols, weird characters like ∣q⟩ instead of writing down an explicit type, etc.). I guess the main advantage is terseness? To me, the mathematical expressions would be so much easier to understand if they were just written in pseudo code or an actual programming language like Python.

Terseness is a significant advantage in pattern recognition. If you write a long, detailed breakdown of every step, not only are you spending a bunch of time writing, you're also obscuring the natural symmetries of the statement. It's like saying "I never understood people who prefer to use functions instead of inlining everything". Adding a bunch of visual noise to a statement doesn't improve comprehension.

Yep. Does anyone think Cauchy's Integral Formula looks worse like this:

https://www.mathjax.org/

than like this?

    z = a + R * sp.exp(sp.I * t)
    
    f_at_a = 1 / (2 * sp.pi * sp.I) * sp.Integral(
        f(z) / (z - a) * sp.diff(z, t),
        (t, 0, 2 * sp.pi),
    )
The heiroglyphs let me offload some of the work to my vision system. It feels like there's some kind of hierarchical attention organization mechanism that works better in visual space. If I'm not pushing to "maximum math," both representations clear the hurdle of "good enough" (CIF does) but if I'm trying to chew through something genuinely gnarly I want all the help I can get.

Re: A walk through of the DeltaNet family of linear attention variants

#119

Post by munksbeer »

abixb wrote: post #81
xp84 wrote: post #66

When I see these types of articles and headlines, it just makes me supremely grateful for all the many people far smarter[1] than me. And humbles me, too, since I actually passed for a "very smart person" in places like high school and undergrad. In fact, I'm 'smart' for an average person, but there are definitely millions of people who make me look like a rube in comparison. [1] I specifically mean those who are abl…

Yes. I continue to believe that humans will still be the source of the vast majority of novel ideas, even as they increasingly use AI-related tools to accelerate their works. One of the though experiments I ran with one of my friends during a recent conversation over drinks was this: raising a bunch of "control group" kids away from the screens and the algorithmic ocean of "normie-tier content," and in a very learner…

>We will always need super smart people with largely unadulterated thinking.

I don't see how you get to that conclusion. We're just atoms. We evolved over a long time and something selected for a particular configuration of atoms that does "something" (see neurons firing in a network) that then somehow produces what you call "novel ideas".

But if one particular set of atoms can do it, so can another. I don't see why we can't eventually invent/evolve/train a bunch of computer atoms to come up with more novel ideas that we can.

Re: A walk through of the DeltaNet family of linear attention variants

#120

Post by edflsafoiewq »

MrFiskarBengt wrote: post #102

You could've also came up with Newton's laws. After all, they look trivial in retrospect. But, there's an important lesson in a story about balancing an egg here that can teach us something. Filippo Brunelleschi said he could build the large dome for the church that had stood unfinished for a century. Skeptical, other's demanded he'd explain how. He refused. Instead he challenged everyone to balance an egg on its tip…

"You could have invented..." a just a genre of expository writing which builds a path from something you know to something you want to learn. The sequence of development is not necessarily even historically accurate, it can be completely invented as long as the learner finds it natural and it helps motivate them.

The fact that invention is hard and actually you probably couldn't have invented a bunch of hard stuff is not really that, um, relevant.

Post reply on HN