Chomsky on what ChatGPT is good for (2023)

> AI this decade have not, at least so far, substantially deepened our understanding of our own intelligence and its constitution

I would push back on this a little bit. While it has not helped us to understand our own intelligence, it has made me question whether such a thing even exists. Perhaps there are no simple and beautiful natural laws, like those that exists in Physics, that can explain how humans think and make decisions. When CNNs learned to recognize faces through a series of hierarchical abstractions that make intuitive sense it's hard to deny the similarities to what we're doing as humans. Perhaps it's all just emergent properties of some messy evolved substrate.

The big lesson from the AI development in the last 10 years from me has been "I guess humans really aren't so special after all" which is similar to what we've been through with Physics. Theories often made the mistake of giving human observers some kind of special importance, which was later discovered to be the cause of theories not generalizing.

"The fact that we have figured out how to translate language into something a computer can "understand" should thrill linguists."

No, there is no understanding at all. Please don't confuse codifying with understanding or translation. LLMs don't understand their input, they simply act on it based on the way they are trained on it.

"And there's a fact here that's very hard to dispute, this method works. I can give a computer instructions and it "understands" them "

No, it really does not understand those instructions. It is at best what used to be called an "idiot savant". Mind you, people used to describe others like that - who is the idiot?

Ask your favoured LLM to write a programme in a less used language - ooh let's try VMware's PowerCLI (it's PowerShell so quite popular) and get it to do something useful. It wont because it can't but it will still spit out something. PowerCLI is not extant across Stackoverflow and co much but it is PS based so the LLMs will hallucinate madder than a hippie on a new super weed.

I think many people are missing the core of what Chomsky is saying. It is often easy to miscommunicate and I think this is primarily what is happening. I think the analogy he gives here really helps emphasize what he's trying to say.

If you're only going to read one part, I think it is this:

  | I mentioned insect navigation, which is an astonishing achievement. Insect scientists have made much progress in studying how it is achieved, though the neurophysiology, a very difficult matter, remains elusive, along with evolution of the systems. The same is true of the amazing feats of birds and sea turtles that travel thousands of miles and unerringly return to the place of origin.

  | Suppose Tom Jones, a proponent of engineering AI, comes along and says: “Your work has all been refuted. The problem is solved. Commercial airline pilots achieve the same or even better results all the time.”

  | If even bothering to respond, we’d laugh.

  | Take the case of the seafaring exploits of Polynesians, still alive among Indigenous tribes, using stars, wind, currents to land their canoes at a designated spot hundreds of miles away. This too has been the topic of much research to find out how they do it. Tom Jones has the answer: “Stop wasting your time; naval vessels do it all the time.”

  | Same response.

It is easy to look at metrics of performance and call things solved. But there's much more depth to these problems than our abilities to solve some task. It's not about just the ability to do something, the how matters. It isn't important that we are able to do better at navigating than birds or insects. Our achievements say nothing about what they do.

This would be like saying we developed a good algorithm only my looking at it's ability to do some task. Certainly that is an important part, and even a core reason for why we program in the first place! But its performance tells us little to nothing about its implementation. The implementation still matters! Are we making good uses of our resources? Certainly we want to be efficient, in an effort to drive down costs. Are there flaws or errors that we didn't catch in our measurements? Those things come at huge costs and fundamentally limit our programs in the first place. The task performance tells us nothing about the vulnerability to hackers nor what their exploits will cost our business.

That's what he's talking about.

Just because you can do something well doesn't mean you have a good understanding. It's natural to think the two relate because understanding improves performance that that's primarily how we drive our education. But this is not a necessary condition and we have a long history demonstrating that. I'm quite surprised this concept is so contentious among programmers. We've seen the follies of using test driven development. Fundamentally, that is the same. There's more depth than what we can measure here and we should not be quick to presume that good performance is the same as understanding[0,1]. We KNOW this isn't true[2].

I agree with Chomsky, it is laughable. It is laughable to think that the man in The Chinese Room[3] must understand Chinese. 40 years in, on a conversation hundreds of years old. Surely we know you can get a good grade on a test without actually knowing the material. Hell, there's a trivial case of just having the answer sheet.

[0] https://www.reddit.com/r/singularity/comments/1dhlvzh/geoffr...

[1] https://www.youtube.com/watch?v=Yf1o0TQzry8&t=449s

[2] https://www.youtube.com/watch?v=hV41QEKiMlM

[3] https://en.wikipedia.org/wiki/Chinese_room

I'm rather confused by this statement. I've read a number of Chomsky pieces and have listened to him speak a number of times. To say his theories were all "not true" seems, to an extent, almost impossible.

Care to expand on how his theories can be taught in such a binary way?

Admittedly my original question (how "not explanatory" leads to "is not a") begins to look like a nit now that I understand the point you were trying to make (or at least I think I do). Nonetheless the discussion seems interesting.

That said, I'm inclined to object to this "explanatory" characteristic you're putting forward. We as humans certainly put a lot of work into optimizing the formulation of our models with the express goal of easing human understanding but I'm not sure that's anything more than an artifact of the system that produces them. At the end of the day they are tools for accomplishing some purpose.

Perhaps the idea you are attempting to express is analogous to concepts such as principal component analysis as applied to the representation of the final model?

> If you have a mechanical clock and quartz-crystal analog clock you are not going to be able to derive the internal workings of either or distinguish between them from the hand positions.

Arguably modern physics analogously does exactly that, although the amount of resources required to do so is astronomical.

Anyhow my claim was not about the ability or lack thereof to derive information from the outputs of a system. It was that as you demand increased accuracy from a model of the hand positions (your example) you will be necessarily forced to model the internal workings of the original physical system to increasingly higher fidelity. I claim that there is no way around this - that fundamentally your only option for increasing the accuracy of the output of a model is for it to more closely resemble the inner workings of the thing being modeled. Taken to the (notably impossible) extreme this might take the form of a quantum mechanics based simulation of the entire system.

Extrapolating this to the weather, I'm claiming that any reasonably accurate ML model will necessarily encompass some sort of underlying truth about the physical system that it is modeling and that as it becomes more accurate it will encode more such truth. Notably, I make no claim about the ability of an unaided human to interpret such truths from a binary blob of weights.

> I don't understand what you mean. Simple models often yield a high level of understanding without being better predictors.

I said nothing about efficiency of educating humans (ie information gathering by or transfer between agents) but rather about model accuracy versus model complexity. I am claiming that more accurate models will invariably be more complex, and that said complexity will invariably encode more information about the original system being modeled. I have yet to encounter a counterexample.

> [CSPRNG recreation]

It is by design impossible to "model" the output of such a function in a bitwise accurate manner without reproducing the internals with perfect fidelity. In the event that someone figures out how to model the output in an imprecise manner without access to the key that would generally be construed as the algorithm having been broken. In other words that example aligns perfectly with my point in the sense that it cannot be approximated to any degree better than random chance with a "simpler" (ie less computationally complex than the original) mechanism. It takes the continuum of accuracy that I was originally describing and replaces it with a step function.

> Yes but if the model does not lead to understanding you cannot come up with the new ideas.

I suppose human understanding is a prerequisite to new human constructed models but my (counter-)point remains. Physics theories are "nothing more" than humans fitting "surface statistics" to increasing degrees of accuracy. I think this is a fairly fundamental truth with regards to the philosophy of science.