Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It's not just statistics: GPT-4 does reason

jbconsulting.substack.com

Post reply on HN
Print view View original

51–60 of 101 posts

Re: It's not just statistics: GPT-4 does reason

#51

Post by eximius »

Part 1: what are n-grams Part 2: it's using embeddings (but a lot of words without actually saying it) Part 3: sufficiently trained NNs can sort things, which isn't statistics

-----

I actually found some of the article interesting but not terribly convincing. Even though I consider these LLMs to be stochastic parrots, that isn't to say they haven't learned something during training, at least according to the colloquial meaning we typically ascribe to even lower models like MNIST classification. I'm even kind of okay with saying that it reasons about things in the same colloquial sense.

In a lot of ways, we just don't have a good definition of what 'reasoning' is. Is it just bad at reasoning because it's input/output/modeling/training is insufficient? Humans struggle to learn multiplication tables when we're young. Are those humans not reasoning because they get the math wrong?

But there isn't plasticity, there isn't adaptability, it's unclear to me that you can effectively inform it how to embed truly novel information - surely something that is possible, with some neurons existing for routing and activating other learned embeddings.

Anyway, interesting stuff.

Re: It's not just statistics: GPT-4 does reason

#52

Post by [deleted] »

eximius wrote: post #51

Part 1: what are n-grams Part 2: it's using embeddings (but a lot of words without actually saying it) Part 3: sufficiently trained NNs can sort things, which isn't statistics ----- I actually found some of the article interesting but not terribly convincing. Even though I consider these LLMs to be stochastic parrots, that isn't to say they haven't learned something during training, at least according to the colloqui…

[deleted]

Re: It's not just statistics: GPT-4 does reason

#53

Post by jbay808 »

eximius wrote: post #51

Part 1: what are n-grams Part 2: it's using embeddings (but a lot of words without actually saying it) Part 3: sufficiently trained NNs can sort things, which isn't statistics ----- I actually found some of the article interesting but not terribly convincing. Even though I consider these LLMs to be stochastic parrots, that isn't to say they haven't learned something during training, at least according to the colloqui…

I'm glad to see someone express this view, because I think this gets to the heart of the question. How does a stochastic parrot learn to sort lists?

Embeddings are part of the compression-by-abstraction that I'm explaining in the first two parts, but the embeddings generated by an LLM go beyond the normal word2vec picture that most people have of embeddings, and I believe are closer to whatever "understanding" means if it could be formally defined. It would be quite a coincidence if GPT-4 happened to solve the riddle merely by virtue of "Moonling" and "cabbage" being closely-located vectors.

Re: It's not just statistics: GPT-4 does reason

#54

Post by alienicecream »

travisjungroth wrote: post #39
physPop wrote: post #30

Earlier quoted context omitted.

I disagree -- that would disprove your claim, as your claim was too broad. Same if they handed you chainsaws or elephants, or seventy-two tennis balls. The more correct claim is you can juggle [some small number of items with particular properties].

Normal English implies that you can do something, not everything. It’s an any versus all distinction, and all is totally unreasonable except for the most formal circumstances. “Can you ride a bike?” “Yeah.” “Prove it. Here I have the world’s smallest bicycle.” <- this person is not worth your time and attention.

It can't even count reliably. And this is a computer, not a human. That is one of the simplest things a computer should be able to do. It can't count because it doesn't know what counting is, not because it's unreliable in the way a human would be when counting. You cannot reason if you do not understand the concepts you are working with. The result is not the measure of success here, because it is good at mimicking, but when it fails at such a basic computing task, you can reasonably conclude it has no idea what it's doing.

Re: It's not just statistics: GPT-4 does reason

#55

Post by jbay808 »

joppy wrote: post #41

The compare-how-big-a-lookup-table-is argument is a bit of a red herring for comparing how complex things are. For example, a 3x3 matrix implements a map from 3 floats to another three floats, a huge space of possibilities (if we have 4-byte floats, this function space has (2^96)^(2^96) elements). From this perspective, representing that map as 9 numbers is an amazing compression ratio. But surely one cannot argue th…

I would interpret this as showing that matrix multiplication code is carefully engineered to correctly implement... well, matrix multiplication. Stumbling on that specific mapping of 96 input bits to 96 output bits would be hard to pick out of a hat by chance, from the set of all possible mappings. Learning that precise mapping, starting from a uniform prior and only given a finite set of examples, could be seen as an impressive task, although less impressive than sorting. If a model learns the correct mapping -- and better yet, needs only 9 parameters to implement it -- then I think it's fairer to say the model does matrix multiplication, rather than that the model convincingly imitates the statistics of matrix multiplication.

Re: It's not just statistics: GPT-4 does reason

#56

Post by sidlls »

shagie wrote: post #38

The Cabbage, Goat, and Wolf problem intrigued me the other day too... so I did a fantasy world. There aren't incompatible pairs but there are limitations on what can be done. Imagine a universe where there are three types of people: wizards, warriors, and priests. Wizards can open a portal that allows two people to go through at a time, but they cannot go through the portal themselves. Priests can summon people from…

That doesn't seem that impressive. The likelihood of reading text having "sphere" in the same context (i.e. within some small number of tokens) as "roll away" is higher than by random chance, because humans have observed and described this behavior in text. There's no indication that GPT4 understands what "roll away" means in any meaningful way: just that it associates the phrase with the word "sphere". It might have chosen "bounce away" instead, and been equally unimpressive.

Re: It's not just statistics: GPT-4 does reason

#57

Post by eximius »

jbay808 wrote: post #53
eximius wrote: post #51

Part 1: what are n-grams Part 2: it's using embeddings (but a lot of words without actually saying it) Part 3: sufficiently trained NNs can sort things, which isn't statistics ----- I actually found some of the article interesting but not terribly convincing. Even though I consider these LLMs to be stochastic parrots, that isn't to say they haven't learned something during training, at least according to the colloqui…

I'm glad to see someone express this view, because I think this gets to the heart of the question. How does a stochastic parrot learn to sort lists? Embeddings are part of the compression-by-abstraction that I'm explaining in the first two parts, but the embeddings generated by an LLM go beyond the normal word2vec picture that most people have of embeddings, and I believe are closer to whatever "understanding" means…

Eh. I still consider them stochastic parrots. My concessions lie elsewhere, primarily in the vocabulary.

We refer to algorithms like quicksort as 'reasoning' about the input. So it's fine to use the same sense of the word to apply to stochastic parrots.

The difference between an LLM learning how to sort things and compiling an implementation of an algorithm like quicksort is not terribly large, from a certain perspective.

I suppose something I'm interested in is whether an LLM that can't sort numbers could be instructed how as a prompt and then do so.

There are some examples of similar phenomenon (the one with some kids made up language was interesting) which suggests the LLMs have a lot of space dedicated towards dynamic pattern selection in their context windows (somewhat tautological) in order to have prompts tune the selection for other layers.

And, of course, lack of plasticity is really interesting.

Re: It's not just statistics: GPT-4 does reason

#58

Post by jbay808 »

eximius wrote: post #57
jbay808 wrote: post #53

Earlier quoted context omitted.

I'm glad to see someone express this view, because I think this gets to the heart of the question. How does a stochastic parrot learn to sort lists? Embeddings are part of the compression-by-abstraction that I'm explaining in the first two parts, but the embeddings generated by an LLM go beyond the normal word2vec picture that most people have of embeddings, and I believe are closer to whatever "understanding" means…

Eh. I still consider them stochastic parrots. My concessions lie elsewhere, primarily in the vocabulary. We refer to algorithms like quicksort as 'reasoning' about the input. So it's fine to use the same sense of the word to apply to stochastic parrots. The difference between an LLM learning how to sort things and compiling an implementation of an algorithm like quicksort is not terribly large, from a certain perspec…

I just can't imagine how a stochastic parrot could repeat back a correctly-sorted list that it hasn't seen in training, without actually implementing a sorting algorithm in the process. It seems (and, by calculation, is) phenomenally unlikely that it would just stochastically happen to pick every single number correctly.

When that is combined with the fact that transformers provably can implement proper deterministic sorting algorithms, it seems that the benefit of the doubt should go to the transformer having learned a sorting algorithm?

LLMs aren't plastic in the sense that they don't learn anything when they aren't being trained. But they can be trained to execute different programs depending on the contents of the context window, like if it contains "wrong, try again:" so maybe they can learn from their mistakes in that sense.

But if you could teach an LLM to sort by explaining it in the context window, the network would already have necessarily learned and stored a sorting algorithm somewhere; the text "here is how sorting is done: [...]" would just be serving as the trigger for that function call.

Re: It's not just statistics: GPT-4 does reason

#59

Post by eximius »

jbay808 wrote: post #58
eximius wrote: post #57

Earlier quoted context omitted.

Eh. I still consider them stochastic parrots. My concessions lie elsewhere, primarily in the vocabulary. We refer to algorithms like quicksort as 'reasoning' about the input. So it's fine to use the same sense of the word to apply to stochastic parrots. The difference between an LLM learning how to sort things and compiling an implementation of an algorithm like quicksort is not terribly large, from a certain perspec…

I just can't imagine how a stochastic parrot could repeat back a correctly-sorted list that it hasn't seen in training, without actually implementing a sorting algorithm in the process. It seems (and, by calculation, is) phenomenally unlikely that it would just stochastically happen to pick every single number correctly. When that is combined with the fact that transformers provably can implement proper deterministic…

Again, I think the disagreement is not whether it has learned to approximate a sorting algorithm, but whether that qualifies as reasoning and, if it does, in what sense.

Re: It's not just statistics: GPT-4 does reason

#60

Post by EGreg »

Imnimo wrote: post #12

>Does the n-gram model really need all those parameters to mimic GPT-4? Yes, it does. I don't understand what this argument is supposed to demonstrate. Obviously you can compress the 8000-gram model that GPT-4 represents - GPT-4's weights are proof!

Does ChatGPT really represent an 8000 gram model? Seems the claim was that it just predicts the next word !
Post reply on HN