There's quite a rationale justification for the ever 'shifting goalposts.' When humans describe some milestone in the future as finally being "real AI", they're not really just describing that milestone, but the many adjacent capabilities that they expect that milestone to entail. But we're quite clever, and like any old metric needing to be juked, we invariably find ways to achieve the milestone while sidestepping all the capabilities it's supposed to entail.
Chess is the obvious example. A machine capable of playing chess at a human level was supposed to indicate the advent of genuine artificial intelligence at one time. It wasn't that playing chess well means one is intelligent, but rather it was assumed it'd entail abstract planning, strategic thought, intuition, and creativity. Of course now we have software which can crush even a world champion, but none of those adjacent capabilities emerged at all.
And so I think it's also increasingly obvious that this is the same thing with chatbots. Many of us thought those 'surrounding capabilities' were finally here, even more so with OpenAI regularly demonstrating exceptional competence on a wide array of distinct metrics, such as performance on the Bar exam. But once you use the system for a while it becomes clear that its knowledge base is absolutely and unbelievably immense, but its 'understanding' of that knowledge is literally zero. It will arbitrarily create e.g. API calls that do not exist, mix up utterly simple concepts, and fail to learn from its mistakes in any meaningful way whatsoever.
I'm sure if you've used ChatGPT for anything you've run into the utterly annoying scenario of:
- "How do I [x]?"
- "Sure! That's easy, just do [A]."
- "No, you're hallucinating."
- "Oh sorry, thanks. You're right you actually need to do [B]!"
- "No, you're still hallucinating."
- "Oh sorry, you're right. You just need to do [A].
If a human, even a stupid human, acted in this way - you'd assume they were trolling you, especially one gifted with the ability for infinite perfect and complete recall.