Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A walk through of the DeltaNet family of linear attention variants

blog.doubleword.ai

Post reply on HN
Print view View original

101–110 of 134 posts

Re: A walk through of the DeltaNet family of linear attention variants

#101

Post by abixb »

phoghed wrote: post #86
abixb wrote: post #81

Earlier quoted context omitted.

Yes. I continue to believe that humans will still be the source of the vast majority of novel ideas, even as they increasingly use AI-related tools to accelerate their works. One of the though experiments I ran with one of my friends during a recent conversation over drinks was this: raising a bunch of "control group" kids away from the screens and the algorithmic ocean of "normie-tier content," and in a very learner…

Not sure I follow, what are you doing with these philosopher kings after you mint them?

Well, the goal was to make them come up with cutting-edge theories and/or hypothesis to the most pressing problems facing humanity. Kids will naturally specialize (just like... MoE models?) as their find their groove while growing up, but the goal is to make sure highly gifted kids from diverse background get highest quality space to think without being moulded or "adulterated" by the noise of pop culture and "everyday" life.

Re: A walk through of the DeltaNet family of linear attention variants

#102

Post by MrFiskarBengt »

You could've also came up with Newton's laws. After all, they look trivial in retrospect. But, there's an important lesson in a story about balancing an egg here that can teach us something.

Filippo Brunelleschi said he could build the large dome for the church that had stood unfinished for a century. Skeptical, other's demanded he'd explain how. He refused. Instead he challenged everyone to balance an egg on its tip. Nobody could do it. He then demonstrated by lightly tapping the egg on the table, flattening the tip, making it stand. "Anyone could've done that! You never said we could break the egg!". And that's the point. Anyone could've done it. But nobody did. Nobody thought 'outside the box'. And likewise, his solution to building the dome is as simple, and as ingenious.

It's called Egg of Columbus. (there's a similar story about Columbus that's more famous, but apparently fictitious). It teaches us that hindsight is 20/20.

Re: A walk through of the DeltaNet family of linear attention variants

#103

Post by brcmthrowaway »

malwrar wrote: post #20
brcmthrowaway wrote: post #8

I could never get this about modern machine/deep learning or even the Transformers. Yes, it's not exactly rocket science, but when I see the data flow diagrams, it's not clear what is calculated in real time or multiple steps. Is it really one big computation f(g(h(x)))?

Yes. Each token prediction is one big function call. Then you just recursively generate more tokens until run out of context or the model predicts a next token indicating end of sequence. Technically the model outputs a matrix where the last row is a probability distribution, but I’m counting sampling from it as part of the chain. Hundreds of billions of dollars has gone into just making the function fatter and gradu…

I remember the concept of layers, as essentially defining the matrix math dimensions. And for a given model/framework, they were static. That always bugged me (not very dynamic).. is this still the case?

Re: A walk through of the DeltaNet family of linear attention variants

#104

Post by abixb »

dmd wrote: post #91
abixb wrote: post #81

Earlier quoted context omitted.

Yes. I continue to believe that humans will still be the source of the vast majority of novel ideas, even as they increasingly use AI-related tools to accelerate their works. One of the though experiments I ran with one of my friends during a recent conversation over drinks was this: raising a bunch of "control group" kids away from the screens and the algorithmic ocean of "normie-tier content," and in a very learner…

You should read 'Anathem' by Neal Stephenson, which goes into great deal about this kind of establishment.

Thank you for the recommendation. I'm yet to checkout any of Neal Stephenson's works, which is real unfortunate.

Re: A walk through of the DeltaNet family of linear attention variants

#105

Post by [deleted] »

olalonde wrote: post #58
TrackerFF wrote: post #19

Machine learning could need, and probably has needed, some unified math notation for the past 15 years IMO. With that said, it was worse back in the day - when ML papers were the products of researchers from all over, you'd see some wild notation. Many will likely disagree with me, but inconsistent notation (across papers!) is to me friction. At least in this article the author explicitly explains the notation at the…

I never understood people who preferred traditional math notation (e.g. single letter symbols, weird characters like ∣q⟩ instead of writing down an explicit type, etc.). I guess the main advantage is terseness? To me, the mathematical expressions would be so much easier to understand if they were just written in pseudo code or an actual programming language like Python.

[deleted]

Re: A walk through of the DeltaNet family of linear attention variants

#106

Post by dr_kretyn »

The bra-ket notation makes this all very simple/intuitive for me. With "vectors" I always get confused which is horizontal/vertical, and then I just follow blobs, and get distracted, and leave. With bra-kets the whole thing was very intuitive! I'm now going to covert other articles to the notation as I must have missed a lot of good stuff!

(Side notes: I have physics PhD and mild dyslexia)

Re: A walk through of the DeltaNet family of linear attention variants

#108

Post by jameshart »

wrs wrote: post #82
jameshart wrote: post #70

Earlier quoted context omitted.

Math educators like Grant Sanderson (3blue1brown) use it in a very specific way: the goal of a mathematical explanation is to make the learner feel like they could have come up with something. And a really good mathematical communicator can absolutely do that. A piece like this which uses it in a headline but in no way makes an average reader feel like they could have come up with it is just badly misjudging how good…

I don’t think “you” in these titles ever really refers to an “average reader”. Some familiarity with the field is required. Imagine how non-programmers (and many programmers) feel about some examples I just Googled: “You Could Have Invented Parser Combinators” “You Could Have Invented Container Runtimes” “You Could Have Invented Git” Given the references to “mathematicians”, I think this reaction is more about an unf…

So I think what’s a little unfair there is that “you could have invented container runtimes”[1] is an article which really tries to answer ‘how does someone who works in traditional software development with an understanding of operating systems and deployment that is a few years out of date get brought up to date on what they need to know so that to them, something like container runtimes become a logical and obvious thing that solves a problem they now understand?’

What this particular mathematical writeup seems to address is ‘how does someone who is already up to speed on the mathematical underpinnings of attention kernel neural networks get brought up to date on what they need to understand in order to see this new algorithm as obvious’

Which is a smaller audience to be sure, and - given how fast this field has grown - somewhat disingenuous in its implications.

This kind of title is appropriate for a piece aimed at helping someone who is a little behind the curve catch up. In the field of deep learning it should be obvious even to the people inside it riding that curve that it is moving really fast and that this kind of content only works for people who are running to keep up.

[1] https://medium.com/@gtrevorjay/you-could-have-invented-conta...

Re: A walk through of the DeltaNet family of linear attention variants

#109

Post by moffkalast »

abixb wrote: post #81
xp84 wrote: post #66

When I see these types of articles and headlines, it just makes me supremely grateful for all the many people far smarter[1] than me. And humbles me, too, since I actually passed for a "very smart person" in places like high school and undergrad. In fact, I'm 'smart' for an average person, but there are definitely millions of people who make me look like a rube in comparison. [1] I specifically mean those who are abl…

Yes. I continue to believe that humans will still be the source of the vast majority of novel ideas, even as they increasingly use AI-related tools to accelerate their works. One of the though experiments I ran with one of my friends during a recent conversation over drinks was this: raising a bunch of "control group" kids away from the screens and the algorithmic ocean of "normie-tier content," and in a very learner…

We should be (and kind of are) doing that with all kids really. I think you've just reinvented the boarding school.

Re: A walk through of the DeltaNet family of linear attention variants

#110

Post by loubbrad »

Asraelite wrote: post #92
TrackerFF wrote: post #19

Machine learning could need, and probably has needed, some unified math notation for the past 15 years IMO. With that said, it was worse back in the day - when ML papers were the products of researchers from all over, you'd see some wild notation. Many will likely disagree with me, but inconsistent notation (across papers!) is to me friction. At least in this article the author explicitly explains the notation at the…

> At least in this article the author explicitly explains the notation at the very start They explain one particular aspect of the notation but never define the variables used. What is k? q? S? It's obvious if you've studied machine learning before, and for some of them you can make an educated guess, but it makes the article mostly opaque if you don't already have some domain-specific background knowledge.

The author does define these terms implicitly at the beginning of §1, where they define standard MHA in this notation. From the formula, (q_t) is the query generated from the hidden state at the (t)-th position (e.g. the token at position (t)), (k_i) is the key for the token at the (i)-th position, (o_t) is the (vector) attention output for the (t)-th position, etc. (S) is then defined later as the sum of the outer products of (k_i) and (v_i) over all positions up to (t). However, I do agree with you that it would not hurt to make this more explicit.
Post reply on HN