Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

State-space models can learn in-context by gradient descent

arxiv.org

Post reply on HN
Print view View original

31–40 of 60 posts

Re: State-space models can learn in-context by gradient descent

#31

Post by quantadev »

littlestymaar wrote: post #30
quantadev wrote: post #29

Earlier quoted context omitted.

If you parse my words a bit more carefully, you'll realize to test my claim there's a simple thought experiment (or real experiment) you can do which is this: Take our "current large size" (my words from last post) LLMs, as they are currently today, and then simply remove the Self-Attention wiring, and see if that destroys the emergent intelligence aspect or not. I claim it would. But at the same time this doesn't me…

You are wildly overestimating the “emergent capabilities” of current models, and underestimate alternative architectures's (namely SSM) performance at the same size. Also, performance of the modern “small” models show that your last sentence isn't really true either.

> wildly overestimating the “emergent capabilities”

How could I be "overestimating" the emergent capabilities when I never even quantified those capabilities other than to call them "emergent" and impressive?

> “small” models show that your last sentence isn't true either.

I never said that even a perfect architecture would make small models "intelligent". However to the extent that even smaller LLMs can exhibit surprising capabilities, that's more evidence IN FAVOR OF everything I've said, not evidence against.

EDIT: But in that last sentence (of prior reply) by "small" what I meant was genuinely small, meaning non-LLM, and you seem to have interpreted it as "a smaller LLM"

Re: State-space models can learn in-context by gradient descent

#32

Post by littlestymaar »

quantadev wrote: post #31
littlestymaar wrote: post #30

Earlier quoted context omitted.

You are wildly overestimating the “emergent capabilities” of current models, and underestimate alternative architectures's (namely SSM) performance at the same size. Also, performance of the modern “small” models show that your last sentence isn't really true either.

> wildly overestimating the “emergent capabilities” How could I be "overestimating" the emergent capabilities when I never even quantified those capabilities other than to call them "emergent" and impressive? > “small” models show that your last sentence isn't true either. I never said that even a perfect architecture would make small models "intelligent". However to the extent that even smaller LLMs can exhibit surp…

Even 1B parameters model show “impressive capabilities” for anyone not accustomed to the current state of the art. And there are plenty of relatively small models that perform as well as ChatGPT 3.5 when it was first released and felt like magic.

“All” that was needed to get there was “just” feeding it more data. The fact that we were actually able to train billion parameters models on multiple trillion tokens is the key property of the transformers, there's no magic beyond that (it's already cool enough though): it's not so much that they are more intelligent, it's simply that with them we can brute-force in a scalable fashion.

Re: State-space models can learn in-context by gradient descent

#33

Post by quantadev »

littlestymaar wrote: post #32
quantadev wrote: post #31

Earlier quoted context omitted.

> wildly overestimating the “emergent capabilities” How could I be "overestimating" the emergent capabilities when I never even quantified those capabilities other than to call them "emergent" and impressive? > “small” models show that your last sentence isn't true either. I never said that even a perfect architecture would make small models "intelligent". However to the extent that even smaller LLMs can exhibit surp…

Even 1B parameters model show “impressive capabilities” for anyone not accustomed to the current state of the art. And there are plenty of relatively small models that perform as well as ChatGPT 3.5 when it was first released and felt like magic. “All” that was needed to get there was “just” feeding it more data. The fact that we were actually able to train billion parameters models on multiple trillion tokens is the…

Yes even the original Transformers model had only millions of parameters and nonetheless showed "impressive capabilities" because it also had Self-Attention.

If you know of any models that have had success (even at the GPT-2 level) without Self-Attention, I'd be interested to know what they are, because I don't know of any.

Re: State-space models can learn in-context by gradient descent

#34

Post by littlestymaar »

quantadev wrote: post #33
littlestymaar wrote: post #32

Earlier quoted context omitted.

Even 1B parameters model show “impressive capabilities” for anyone not accustomed to the current state of the art. And there are plenty of relatively small models that perform as well as ChatGPT 3.5 when it was first released and felt like magic. “All” that was needed to get there was “just” feeding it more data. The fact that we were actually able to train billion parameters models on multiple trillion tokens is the…

Yes even the original Transformers model had only millions of parameters and nonetheless showed "impressive capabilities" because it also had Self-Attention. If you know of any models that have had success (even at the GPT-2 level) without Self-Attention, I'd be interested to know what they are, because I don't know of any.

RWKV.

There aren't many multi-billion-parameters non-transformer models because of path dependence, but that doesn't mean that only transformers can achieve this kind of results.

Re: State-space models can learn in-context by gradient descent

#35

Post by quantadev »

littlestymaar wrote: post #34
quantadev wrote: post #33

Earlier quoted context omitted.

Yes even the original Transformers model had only millions of parameters and nonetheless showed "impressive capabilities" because it also had Self-Attention. If you know of any models that have had success (even at the GPT-2 level) without Self-Attention, I'd be interested to know what they are, because I don't know of any.

RWKV. There aren't many multi-billion-parameters non-transformer models because of path dependence, but that doesn't mean that only transformers can achieve this kind of results.

My statements (which you disagreed with, without exception) haven't been about Transformers v.s. non-Transformers. Everything above has been about the importance of the Self-Attention part of it. We could remove Self-Attention from Transformers and still have a functional (but dumb) NN, and that was my point.

Your position was that the Self-Attention is a less important part (because UAT, yadda yadda), and my position was that it's the key ingredient. Every statement above that I made, that you called wrong, was correct. lol.

Re: State-space models can learn in-context by gradient descent

#36

Post by littlestymaar »

quantadev wrote: post #35
littlestymaar wrote: post #34

Earlier quoted context omitted.

RWKV. There aren't many multi-billion-parameters non-transformer models because of path dependence, but that doesn't mean that only transformers can achieve this kind of results.

My statements (which you disagreed with, without exception) haven't been about Transformers v.s. non-Transformers. Everything above has been about the importance of the Self-Attention part of it. We could remove Self-Attention from Transformers and still have a functional (but dumb) NN, and that was my point. Your position was that the Self-Attention is a less important part (because UAT, yadda yadda), and my positio…

You are moving the goalpost. The discussion has always been about transformers vs non transformers.

You claimed that self attention was needed to achieve the level of intelligence that we've seen with GPT 3.5:

> without those attention heads even the scaling up to current parameter sizes we have to day would not have ended up with the level of emergent intelligence that shocked the world with GPT 3.5. (Verbatim quote from you https://news.ycombinator.com/item?id=41986010)

This is the claim I've been disputing, by responding that the key feature of the intelligence of tranformer models come from their scalability. And now that we have alternative that scale equally well (SSM and RWKV) unsurprisingly we see them achieve the same level of reasoning abilities.

> Every statement above that I made, that you called wrong, was correct. lol.

Well, except the one quoted above at least…

Re: State-space models can learn in-context by gradient descent

#37

Post by quantadev »

littlestymaar wrote: post #36
quantadev wrote: post #35

Earlier quoted context omitted.

My statements (which you disagreed with, without exception) haven't been about Transformers v.s. non-Transformers. Everything above has been about the importance of the Self-Attention part of it. We could remove Self-Attention from Transformers and still have a functional (but dumb) NN, and that was my point. Your position was that the Self-Attention is a less important part (because UAT, yadda yadda), and my positio…

You are moving the goalpost. The discussion has always been about transformers vs non transformers. You claimed that self attention was needed to achieve the level of intelligence that we've seen with GPT 3.5: > without those attention heads even the scaling up to current parameter sizes we have to day would not have ended up with the level of emergent intelligence that shocked the world with GPT 3.5. (Verbatim quote…

In the quote you're calling wrong (41986010), you're interpreting "scaling up" as "scaling up, including changing architecture". Scaling up transformers just means scaling up transformers, and keeping everything else the same. In other words you're interpreting "parameter size" as "parameter size, independent of architecture", and I meant parameter size of a Transformer (in the context of with v.s. without Self-Attention).

Re: State-space models can learn in-context by gradient descent

#38

Post by littlestymaar »

quantadev wrote: post #37
littlestymaar wrote: post #36

Earlier quoted context omitted.

You are moving the goalpost. The discussion has always been about transformers vs non transformers. You claimed that self attention was needed to achieve the level of intelligence that we've seen with GPT 3.5: > without those attention heads even the scaling up to current parameter sizes we have to day would not have ended up with the level of emergent intelligence that shocked the world with GPT 3.5. (Verbatim quote…

In the quote you're calling wrong (41986010), you're interpreting "scaling up" as "scaling up, including changing architecture". Scaling up transformers just means scaling up transformers, and keeping everything else the same. In other words you're interpreting "parameter size" as "parameter size, independent of architecture", and I meant parameter size of a Transformer (in the context of with v.s. without Self-Atten…

Pathetic.

Re: State-space models can learn in-context by gradient descent

#39

Post by quantadev »

littlestymaar wrote: post #38
quantadev wrote: post #37

Earlier quoted context omitted.

In the quote you're calling wrong (41986010), you're interpreting "scaling up" as "scaling up, including changing architecture". Scaling up transformers just means scaling up transformers, and keeping everything else the same. In other words you're interpreting "parameter size" as "parameter size, independent of architecture", and I meant parameter size of a Transformer (in the context of with v.s. without Self-Atten…

Pathetic.

Straw-manning failed, so now you insult.

Re: State-space models can learn in-context by gradient descent

#40

Post by littlestymaar »

quantadev wrote: post #39
littlestymaar wrote: post #38

Earlier quoted context omitted.

Pathetic.

Straw-manning failed, so now you insult.

There's no staw-man, and you are now at the point of trying to re-invent the definition of words in order to somehow “win the argument ” without even respecting your own previous position. This behavior is legit pathetic, it's not an insult it's a fact. Respect yourself.
Post reply on HN