Chomsky on what ChatGPT is good for (2023)

But even so,

> In other words, I would claim that as a model more closely matches those "surface statistics" it necessarily more closely resembles the underlying mechanisms that gave rise to them.

I don't what it means, for example, for a deep neural network, to "more resemble" the underlying process of the weather. It's also obviously false in general: If you have a mechanical clock and quartz-crystal analog clock you are not going to be able to derive the internal workings of either or distinguish between them from the hand positions. The same is true for two different pseudo-random number generator circuits that produce the same output.

> I have yet to see an example where a more accurate model was conceptually simpler than the simplest known model at some lower level of accuracy.

I don't understand what you mean. Simple models often yield a high level of understanding without being better predictors. For example an idealized ball rolling down a plane, Galileo's mass/gravity thought experiment, Kepler etc. Many of these models ignore less important details to focus on the fundamental ones.

> From an information theoretic angle I think it's similar to compression (something that ML also happens to be almost unbelievably good at). Related to this, I've seen it argued somewhere (I don't immediately recall where though) that learning (in both the ML and human sense) amounts to constructing a world model via compression and that rings true to me.

In practice you get nowhere trying to recreate the internals of a cryptographic pseudo-random number generator from the output it produces (maybe in theory you could do it with infinite data and no bounds on computational complexity or something) even though the generator itself could be highly compressed.

> Sure, but what leads to those theories? They are invariably the result of attempting to more accurately model the things which we can observe.

Yes but if the model does not lead to understanding you cannot come up with the new ideas.

It certainly teaches us many things. But an LLM trained on as many words (or generally speaking an AI trained on sounds) in similar quantities of a toddler learning to understand, parse and apply language, would not perform well with current architectures. They need orders of magnitude more training material to get even close. Basically, current AI learns slowly, but of course it’s much faster in wall clock time because it’s all computer.

What I mean is: what makes an ALU (CPU) better than a human at arithmetic? It’s just faster and makes fewer errors. Similarly, what makes Google or Wikipedia better than an educated person? It’s just storing and helping you access stored information, it’s not magic (anymore). You can manually do everything mechanically, if you’re willing to waste the time to prove a point.

An LLM does many things better than humans, but we forget they’ve been trained on all written history and have hundreds of billions of parameters. If you compare what an LLM can do with the same amount of training to a human, the human is much better even at picking up patterns – current AIs strongest skill. The magic comes from the unseen vast amounts of training data. This is obvious when using them – stray just slightly outside of the training zone to unfamiliar domains and ”ability” drops rapidly. The hard part is figuring out these fuzzy boundaries. How far does interpolating training data get you? What are the highest level patterns are encoded in the training data? And most importantly, to what extent do those patterns apply to novel domains?

Alternatively, you can use LLMs as a proxy for understanding the relationship between domains, instead of letting humans label them and decide the taxonomy. One such example is the relationship between detecting patterns and generating text and images – it turns out to be more or less reversible through the same architecture. More such remarkable similarities and anti-similarities are certainly on the horizon. For instance, my gut feeling says that small talk is closer to driving a car but very different from puzzle solving. We don’t really have a (good) taxonomy over human- or animal brain processes.

So what is it really gonna do with a book, that LLM ? Reorder its internal matrix to be a little bit more precise when autocompleting sentences sounding like the book ? We could build an nvidia cluster the size of the Sun and it would repeat sentences back to us in unbelievable ways but would still be unable to take a knowledge-based decision, I fear.

So what are we in awe at exactly ? A pretty parrot.

The day the Chinese room metaphor disappears is when ChatGPT replies to you that your question is so boring it doesn't want to expend the resources to think about it. But it'd be ready to talk about this or that, that it's currently trying to get better at. When it finally has agency over its own intelligence. When it acquires a purpose.

Who would make such a claim? LLM’s are of course incredible, but it seems obvious that their mechanism is quite different than the human brain.

I think the best you can say is that one could motivate lines of inquiry in human understanding, especially because we can essentially do brain surgery on an LLM in action in a way that we can’t with humans.