Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

State-space models can learn in-context by gradient descent

arxiv.org

Post reply on HN
Print view View original

11–20 of 60 posts

Re: State-space models can learn in-context by gradient descent

#11

Post by knowaveragejoe »

derefr wrote: post #7

So, I'm just a layman when it comes to AI/ML, but I do understand computability — what's possible to do with a given machine, and how we can build higher-computational-power primitives out of lower-computational-power primitives by plugging those primitives together with "glue" like parallel feed-forward chains (e.g. an ALU adder's carry bits) and loops over static sub-states of execution. My own mental model for wha…

Your intuition is, I think, pretty close to accurate. See this paper from earlier this year:

> While Transformers have been the main architecture behind deep learning's success in language modeling, state-space models (SSMs) such as Mamba have recently been shown to match or outperform Transformers at small to medium scale. We show that these families of models are actually quite closely related, and develop a rich framework of theoretical connections between SSMs and variants of attention, connected through various decompositions of a well-studied class of structured semiseparable matrices. Our state space duality (SSD) framework allows us to design a new architecture (Mamba-2) whose core layer is an a refinement of Mamba's selective SSM that is 2-8X faster, while continuing to be competitive with Transformers on language modeling.

https://arxiv.org/abs/2405.21060

Re: State-space models can learn in-context by gradient descent

#12

Post by techbro92 »

quantadev wrote: post #6

> can reproduce the outputs of an implicit linear model with least squares loss after one step of gradient descent. Makes you wonder if we're training LLMs the hard way. For example, if computers had been invented before Calculus, we'd have been using "Numerical Integration" (iterating the differential squares to sum up areas, etc) and "Numerical Differentiation" (ditto for calculating slopes). So I wonder if we're s…

There are actually neural networks with explicit optimization layers but I don’t think these have really had much success.

Re: State-space models can learn in-context by gradient descent

#13

Post by rachofsunshine »

kbr wrote: post #9
quantadev wrote: post #6

> can reproduce the outputs of an implicit linear model with least squares loss after one step of gradient descent. Makes you wonder if we're training LLMs the hard way. For example, if computers had been invented before Calculus, we'd have been using "Numerical Integration" (iterating the differential squares to sum up areas, etc) and "Numerical Differentiation" (ditto for calculating slopes). So I wonder if we're s…

NNs have complex non-convex loss functions that don't admit a closed-form solution. Even for small models, it can be shown that it's an NP-complete problem. In fact, even for linear regression (least squares), which has a closed-form solution, it can be computationally cheaper to run gradient descent since finding the closed form solution requires you to calculate and invert a large matrix (X^T X).

Which in some sense is intuitive: any closed form that can model general computation to any significant degree should be hard: if it weren't, you could encode your NP-complete problem into it, solve it in an efficient closed form, and collect your Fields medal for proving P = NP.

Re: State-space models can learn in-context by gradient descent

#14

Post by quantadev »

rachofsunshine wrote: post #13
kbr wrote: post #9

Earlier quoted context omitted.

NNs have complex non-convex loss functions that don't admit a closed-form solution. Even for small models, it can be shown that it's an NP-complete problem. In fact, even for linear regression (least squares), which has a closed-form solution, it can be computationally cheaper to run gradient descent since finding the closed form solution requires you to calculate and invert a large matrix (X^T X).

Which in some sense is intuitive: any closed form that can model general computation to any significant degree should be hard: if it weren't, you could encode your NP-complete problem into it, solve it in an efficient closed form, and collect your Fields medal for proving P = NP.

Intuition is often wrong, even for high IQ people, like your average HN user. lol.

For a long time it was intuitive that you cannot find the area under arbitrary functions, but then Calculus was invented, showing us a new "trick", that was previously unfathomable, and indistinguishable from magic.

I'm just not sure mankind's understanding of Mathematics is out of new "tricks" to be learned. I think there are types of algorithms today that look like the require N-iterations to get X-precision, when in reality we might be able to divide N by some factor, for some algorithms, and still end up with X-precision.

Re: State-space models can learn in-context by gradient descent

#15

Post by quantadev »

techbro92 wrote: post #12
quantadev wrote: post #6

> can reproduce the outputs of an implicit linear model with least squares loss after one step of gradient descent. Makes you wonder if we're training LLMs the hard way. For example, if computers had been invented before Calculus, we'd have been using "Numerical Integration" (iterating the differential squares to sum up areas, etc) and "Numerical Differentiation" (ditto for calculating slopes). So I wonder if we're s…

There are actually neural networks with explicit optimization layers but I don’t think these have really had much success.

I just have a hunch we're in early days still even with Transformers architectures. The MLP (Perceptron) is such a simple mathematical structure and mostly doing linear stuff (tons of multiplications, then a few adds, and a squashing-type activation function), plus the attention heads add-on from the Transformers paper too, of course (and other minor things) but ultimately it's a very easy to understand data structure so it's hard for me to believe there's not massive leaps and bounds that we can take to gain orders of magnitude more performance just like the leap that the Transformers paper had.

Re: State-space models can learn in-context by gradient descent

#16

Post by littlestymaar »

quantadev wrote: post #15
techbro92 wrote: post #12

Earlier quoted context omitted.

There are actually neural networks with explicit optimization layers but I don’t think these have really had much success.

I just have a hunch we're in early days still even with Transformers architectures. The MLP (Perceptron) is such a simple mathematical structure and mostly doing linear stuff (tons of multiplications, then a few adds, and a squashing-type activation function), plus the attention heads add-on from the Transformers paper too, of course (and other minor things) but ultimately it's a very easy to understand data structur…

> We can take to gain orders of magnitude more performance just like the leap that the Transformers paper had.

Afaik the most important benefit of transformers aren't their “performance” (in the sense of ability to perform their tasks) but their scalability which come from their ability to be trained and evaluated efficiently on big GPU clusters, which isn't something you can do with recurrent neural networks.

And then, if I understood correctly, the benefit of state-space models being that you can train them in parallel and run them in a recurrent fashion, making inference cheaper than transformers especially when context size grow.

Re: State-space models can learn in-context by gradient descent

#17

Post by taneq »

quantadev wrote: post #6

> can reproduce the outputs of an implicit linear model with least squares loss after one step of gradient descent. Makes you wonder if we're training LLMs the hard way. For example, if computers had been invented before Calculus, we'd have been using "Numerical Integration" (iterating the differential squares to sum up areas, etc) and "Numerical Differentiation" (ditto for calculating slopes). So I wonder if we're s…

There's a whole lotta certainty about even intractable integrals which is lacking in the case of neural nets grappling with noisy incomplete real world data.

Re: State-space models can learn in-context by gradient descent

#18

Post by radarsat1 »

eli_gottlieb wrote: post #8

>Our key insight is that the diagonal linear recurrent layer can act as a gradient accumulator So they're sort of reinventing the discrete-time differentiator from signal processing, but parameterized neurally?

Converging slowly on Kalman filters, calling it now.

Re: State-space models can learn in-context by gradient descent

#19

Post by RaftPeople »

quantadev wrote: post #14
rachofsunshine wrote: post #13

Earlier quoted context omitted.

Which in some sense is intuitive: any closed form that can model general computation to any significant degree should be hard: if it weren't, you could encode your NP-complete problem into it, solve it in an efficient closed form, and collect your Fields medal for proving P = NP.

Intuition is often wrong, even for high IQ people, like your average HN user. lol. For a long time it was intuitive that you cannot find the area under arbitrary functions, but then Calculus was invented, showing us a new "trick", that was previously unfathomable, and indistinguishable from magic. I'm just not sure mankind's understanding of Mathematics is out of new "tricks" to be learned. I think there are types of…

> I'm just not sure mankind's understanding of Mathematics is out of new "tricks" to be learned.

This is my opinion also as it relates to AI/ANN. Things I read about how scientists see the brain shifting due to learning (minimum energy of network type stuff) seem like the brain has some functions figured out that we haven't identified yet.

Maybe it's math already fully understood just not applied well to ANN's, but maybe there's some secret sauce in there.

Re: State-space models can learn in-context by gradient descent

#20

Post by quantadev »

littlestymaar wrote: post #16
quantadev wrote: post #15

Earlier quoted context omitted.

I just have a hunch we're in early days still even with Transformers architectures. The MLP (Perceptron) is such a simple mathematical structure and mostly doing linear stuff (tons of multiplications, then a few adds, and a squashing-type activation function), plus the attention heads add-on from the Transformers paper too, of course (and other minor things) but ultimately it's a very easy to understand data structur…

> We can take to gain orders of magnitude more performance just like the leap that the Transformers paper had. Afaik the most important benefit of transformers aren't their “performance” (in the sense of ability to perform their tasks) but their scalability which come from their ability to be trained and evaluated efficiently on big GPU clusters, which isn't something you can do with recurrent neural networks. And th…

The biggest thing I had understood about the Transformers Paper (Attention is all you Need) is how the "attention heads" vectors are wired up in such a way as to allow words to be "understood" in the proper context. In other words "see spot run" is different from "run a computer program" has dramatically different but specific context for the word "run".

It was also my understanding that without those attention heads even the scaling up to current parameter sizes we have to day would not have ended up with the level of emergent intelligence that shocked the world with GPT 3.5. We needed both very large models and words put into semantic context in semantic space.

Post reply on HN