Don't believe the hype: why ChatGPT is not the “holy grail” of AI research

"Successful technology was once argued against" is not evidence that "any technology that is argued against will be successful".

That should not be a hard concept to grasp, yet it seems to elude so many. A -> B does not mean B -> A. That's literally like day two of any class on formal logic, and only because the first day was spent discussing the syllabus.

LLMs act in insensible ways all the time. They contradict themselves. They hallucinate. If you ask an LLM to follow a simple but long logic puzzle and show you its work, it will often make extremely obvious errors and fail to notice even when you ask it to review its work.

What LLMs can do is coherently string together language. That requires sophisticated linguistic understanding, and LLMs are pretty impressive for it. But merely understanding language is not intelligence, or even common sense. I suspect we're approaching the limits of what we can get out of statistical language generation.

Actual AGI would require a logical model of the world, not a probabilistic model of language. Looking at non-human animals, we can see problem-solving evolved long before language did. Language is a second-order phenomenon we use to express first-order problem-solving conclusions about the world, and I'm skeptical that we'll ever manage to accurately recreate first-order problem-solving by training models on the second-order linguistic artefacts of problem-solving. It's actually very easy to create superficially-realistic second-order problem-solving artefacts which, when examined with first-order problem solving capabilities, don't stand up to scrutiny—i.e. contradictions, hallucinations, and faulty reasoning. I suspect that, when computers do learn to problem-solve, it will have been by training to solve problems.

There's also no way we'll get the kind of growth you're predicting from self-play. When it comes to a simple competition like a board game, it's easy to optimize for more skilled play. But there's no objective way to win a conversation. Maybe we'll get some gains out of training these systems against each other, but it's just not a clearly viable use case.

In my understanding, LLMs already hit a big wall. We can't increase the size of models mainly because it's too expensive, but also doing so may not be as effective as before. We've also run out of data. The free lunch is likely already over, for now. It's unlikely that we'll see huge improvements in the direction we've seen during recent years.

Instead, what I see is that the first letter 'L' is getting smaller. People are working on (relatively) smaller specialized models. But it means these models are unlikely outperform larger LLMs (in the direction mentioned above).

I think ChatGPT and the likes should be thought of as cartographers who when probed can provide an outline of regions of the knowledge space and give hints as to what could be useful.

Given then a high level but also imprecise map, I can ask the GPS for clarity as to where I am going.

The problem is that I often don't know where that is and thus I don't know what to ask from the GPS, but I can describe it to the cartographer, and they can draw a map for me.

In this analogy, the GPS is search engines / books / papers.

If you're not an expert in your field, GPT will successfully lie to you. There's no second opinion.

Rather than a feelings filter, it needs an integrity filter.