Two common cognitive errors to beware of when reasoning about the current state of AI/LLM this exhibits:
1. reasoning by inappropriate/incomplete analogy
It is not accurate (predictive) to describe what these systems do as mimicking or regurgitating human output, or, e.g. describing what they do with reference to Markov chains and stochastic outcomes.
This is increasingly akin to using the same overly reductionist framing of what humans do, and loses any predictive ability at all.
To put a point on it, this line of critique conflates things like agency and self-awareness, with other tiers of symbolic representation and reasoning about the world hitherto reserved to humans. These systems build internal state and function largely in terms of analogical reasoning themselves.
This is a lot more that "mimickery" regardless of their lack of common sense.
2. assuming stasis and failure to anticipate non-linearities and punctured equilibrium
The last thing these systems are is in their final form. What exists as consumer facing scaled product is naturally generationally behind what is in beta, or alpha; and one of the surprises (including to those of us in the industry...) of these systems is the extent to which behaviors emerge.
Whenever you find yourself thinking, "AI is never going to..." you can stop the sentence, because it's if not definitionally false, quite probably false.
None of us know where we are in the so-called sigmoid curve, but it is already clear we are far from reaching any natural asymptotes.
A pertinent example of this is to go back a year and look at the early output of e.g. Midjourney, and the prompt engineering that it took to produce various images; and compare that with the state of the (public-facing) art today... and to look at the failure of anyone (me included) to predict just how quickly things would advance.
Our hands are now off the wheel. We just might have a near-life experience.