Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

State-space models can learn in-context by gradient descent

arxiv.org

Post reply on HN
Print view View original

21–30 of 60 posts

Re: State-space models can learn in-context by gradient descent

#21

Post by quantadev »

RaftPeople wrote: post #19
quantadev wrote: post #14

Earlier quoted context omitted.

Intuition is often wrong, even for high IQ people, like your average HN user. lol. For a long time it was intuitive that you cannot find the area under arbitrary functions, but then Calculus was invented, showing us a new "trick", that was previously unfathomable, and indistinguishable from magic. I'm just not sure mankind's understanding of Mathematics is out of new "tricks" to be learned. I think there are types of…

> I'm just not sure mankind's understanding of Mathematics is out of new "tricks" to be learned. This is my opinion also as it relates to AI/ANN. Things I read about how scientists see the brain shifting due to learning (minimum energy of network type stuff) seem like the brain has some functions figured out that we haven't identified yet. Maybe it's math already fully understood just not applied well to ANN's, but m…

One reason to believe there's even new low hanging fruit (that doesn't even require new math) is how simple and trivial the "Attention Heads" structure of the Transformer architecture really is. It's not advanced at all. It was just a great ideal that panned out that pretty much any creative AI researcher could've thought up after smokin' a joint. lol. I mean someone could do trivial experiments with different Perceptron network structuring and end up revolutionizing the world.

I think things are gonna get interesting real quick once LLMs themselves start "self experimenting" with writing code for different architectures.

Re: State-space models can learn in-context by gradient descent

#22

Post by quantadev »

taneq wrote: post #17
quantadev wrote: post #6

> can reproduce the outputs of an implicit linear model with least squares loss after one step of gradient descent. Makes you wonder if we're training LLMs the hard way. For example, if computers had been invented before Calculus, we'd have been using "Numerical Integration" (iterating the differential squares to sum up areas, etc) and "Numerical Differentiation" (ditto for calculating slopes). So I wonder if we're s…

There's a whole lotta certainty about even intractable integrals which is lacking in the case of neural nets grappling with noisy incomplete real world data.

There's at least 100 different equally likely interpretations of that particular sequence of words you just wrote.

Re: State-space models can learn in-context by gradient descent

#23

Post by agnosticmantis »

These papers don’t explain how pertained LLMs learn in-context, because the simplified models in these papers are either pretrained for the same task that’s tested in-context, or the weights are handpicked by humans to do GD at inference time.

See this video for a good discussion: https://youtu.be/-yo2672UikU

Re: State-space models can learn in-context by gradient descent

#24

Post by littlestymaar »

quantadev wrote: post #20
littlestymaar wrote: post #16

Earlier quoted context omitted.

> We can take to gain orders of magnitude more performance just like the leap that the Transformers paper had. Afaik the most important benefit of transformers aren't their “performance” (in the sense of ability to perform their tasks) but their scalability which come from their ability to be trained and evaluated efficiently on big GPU clusters, which isn't something you can do with recurrent neural networks. And th…

The biggest thing I had understood about the Transformers Paper (Attention is all you Need) is how the "attention heads" vectors are wired up in such a way as to allow words to be "understood" in the proper context. In other words "see spot run" is different from "run a computer program" has dramatically different but specific context for the word "run". It was also my understanding that without those attention heads…

Attention heads existed before Transformers, they where used in recurrent neural networks (RNN) to improve their performance. The paper is called “Attention is all you need” because transformers keep the attention head while discarding the RNN part entirely.

Getting rid of RNN vastly improved training scalability and allowed big players to start training enormous models on even more enormous training set in ways that weren't possible with a RNN AFAIK.

Re: State-space models can learn in-context by gradient descent

#25

Post by quantadev »

littlestymaar wrote: post #24
quantadev wrote: post #20

Earlier quoted context omitted.

The biggest thing I had understood about the Transformers Paper (Attention is all you Need) is how the "attention heads" vectors are wired up in such a way as to allow words to be "understood" in the proper context. In other words "see spot run" is different from "run a computer program" has dramatically different but specific context for the word "run". It was also my understanding that without those attention heads…

Attention heads existed before Transformers, they where used in recurrent neural networks (RNN) to improve their performance. The paper is called “Attention is all you need” because transformers keep the attention head while discarding the RNN part entirely. Getting rid of RNN vastly improved training scalability and allowed big players to start training enormous models on even more enormous training set in ways that…

When discussing "Attention Heads" in the context of the Transformers Paper, there's no need to put the word "Self" in front of it, as in "Self-Attention". That's the context in which I used the word Attention above. Something similar to self-attention had pre-existed this paper, but not actual self-attention.

You're right that getting rid of "Recurrence" was another innovation, but removing it was probably more of a hack to make things parallelizable, than something that was architecturally justifiable from first principles (like self-attention is), because there's definite "power" in Recurrence (making it desirable), but it's just too costly to run that in LLMs because of CPU cycles.

Re: State-space models can learn in-context by gradient descent

#26

Post by littlestymaar »

quantadev wrote: post #25
littlestymaar wrote: post #24

Earlier quoted context omitted.

Attention heads existed before Transformers, they where used in recurrent neural networks (RNN) to improve their performance. The paper is called “Attention is all you need” because transformers keep the attention head while discarding the RNN part entirely. Getting rid of RNN vastly improved training scalability and allowed big players to start training enormous models on even more enormous training set in ways that…

When discussing "Attention Heads" in the context of the Transformers Paper, there's no need to put the word "Self" in front of it, as in "Self-Attention". That's the context in which I used the word Attention above. Something similar to self-attention had pre-existed this paper, but not actual self-attention. You're right that getting rid of "Recurrence" was another innovation, but removing it was probably more of a…

> removing it was probably more of a hack to make things parallelizable

But that's the entire point of it. Transformer-based LLM are “more intelligent” just because you can make them bigger and train them on bigger datasets because of this parallelization.

Re: State-space models can learn in-context by gradient descent

#27

Post by quantadev »

littlestymaar wrote: post #26
quantadev wrote: post #25

Earlier quoted context omitted.

When discussing "Attention Heads" in the context of the Transformers Paper, there's no need to put the word "Self" in front of it, as in "Self-Attention". That's the context in which I used the word Attention above. Something similar to self-attention had pre-existed this paper, but not actual self-attention. You're right that getting rid of "Recurrence" was another innovation, but removing it was probably more of a…

> removing it was probably more of a hack to make things parallelizable But that's the entire point of it. Transformer-based LLM are “more intelligent” just because you can make them bigger and train them on bigger datasets because of this parallelization.

It's not just about size. Self-Attention is every bit as important as large size, because if we had the current large size, but without Self-Attention we wouldn't have the emergent intelligence. Also "size" isn't even a new innovation. Self-Attention was a new innovation.

Re: State-space models can learn in-context by gradient descent

#28

Post by littlestymaar »

quantadev wrote: post #27
littlestymaar wrote: post #26

Earlier quoted context omitted.

> removing it was probably more of a hack to make things parallelizable But that's the entire point of it. Transformer-based LLM are “more intelligent” just because you can make them bigger and train them on bigger datasets because of this parallelization.

It's not just about size. Self-Attention is every bit as important as large size, because if we had the current large size, but without Self-Attention we wouldn't have the emergent intelligence. Also "size" isn't even a new innovation. Self-Attention was a new innovation.

This doesn't match with the common knowledge on the topic, which is that model size is more important than the architecture. And training size is even more important, which is why single digit billion parameters are strongers than hundreds-of-billion ones from several years early when “Chinchilla optimal training” was in fashion.

SSM are literally the proof that all that really matters is training scalability.

The Universal approximation theorem doesn't care about the architecture after all.

Re: State-space models can learn in-context by gradient descent

#29

Post by quantadev »

littlestymaar wrote: post #28
quantadev wrote: post #27

Earlier quoted context omitted.

It's not just about size. Self-Attention is every bit as important as large size, because if we had the current large size, but without Self-Attention we wouldn't have the emergent intelligence. Also "size" isn't even a new innovation. Self-Attention was a new innovation.

This doesn't match with the common knowledge on the topic, which is that model size is more important than the architecture. And training size is even more important, which is why single digit billion parameters are strongers than hundreds-of-billion ones from several years early when “Chinchilla optimal training” was in fashion. SSM are literally the proof that all that really matters is training scalability. The Un…

If you parse my words a bit more carefully, you'll realize to test my claim there's a simple thought experiment (or real experiment) you can do which is this:

Take our "current large size" (my words from last post) LLMs, as they are currently today, and then simply remove the Self-Attention wiring, and see if that destroys the emergent intelligence aspect or not. I claim it would. But at the same time this doesn't mean you can just stick Self-Attention onto a small model and expect intelligence to once again emerge.

Re: State-space models can learn in-context by gradient descent

#30

Post by littlestymaar »

quantadev wrote: post #29
littlestymaar wrote: post #28

Earlier quoted context omitted.

This doesn't match with the common knowledge on the topic, which is that model size is more important than the architecture. And training size is even more important, which is why single digit billion parameters are strongers than hundreds-of-billion ones from several years early when “Chinchilla optimal training” was in fashion. SSM are literally the proof that all that really matters is training scalability. The Un…

If you parse my words a bit more carefully, you'll realize to test my claim there's a simple thought experiment (or real experiment) you can do which is this: Take our "current large size" (my words from last post) LLMs, as they are currently today, and then simply remove the Self-Attention wiring, and see if that destroys the emergent intelligence aspect or not. I claim it would. But at the same time this doesn't me…

You are wildly overestimating the “emergent capabilities” of current models, and underestimate alternative architectures's (namely SSM) performance at the same size.

Also, performance of the modern “small” models show that your last sentence isn't really true either.

Post reply on HN