This is a great example when it comes to trying to understand the epistemological limits of AI. People inherently fall back on an argument that presumes the human mind works like a neural network.
There's an interesting theory that I've never been able to identify the origins of, that humans like to think that we created technology from needs based on our models of the world, whereas we ignore the effects of daily technology on our thinking frameworks. The argument essentially states that we never "discovered" the circulatory system of the heart or "discovered" it works like a pump and valves, rather instead right around the same time this theoretical work was being investigated on the heart is when the industrial revolution was in full swing. Thus, we modelled the heart as pumps and valves because that's the technology we were surrounded by. The heart isn't somehow inherently a "pump" and we "discovered" that, we just started using the pump metaphor because it seemed to help do other things. I'm sure though that the metaphor has it limits.
Typically, the narrative around the invention of machine learning models is that we started coding computers to be more like recently "discovered" models of the brain.
Under this theory its the opposite. Right as cognitive sciences started developing as a novel field of research is when we developed computers. So, in classic form we decomposed the brain in atomic fashion, and the 'atoms' of measurement we chose to use ended up being bits and bytes.
Distinguishing between inventing "novel" things and gobbledygook is completely subjective and based on the viewer's own models. It's proving these abilities after the fact, not before it. Thousand monkeys on typewriters etc.
This measurement of "accuracy" is completely forgetting everything that Kuhn discovered about scientific knowledge. If you've got a community of like 3 people who research some incredibly esoteric scientific field, only those 3 people could ever accurately judge the full extent of their domain. A model could generate a series of tokens that for the rest of the world is gobbledygook, but to these 3 scientists it makes perfect sense. This doesn't really endeavour me to believe that there's anything "novel" about what AI "predicts". It just throws out enough combinations and we conveniently ignore the huge gaps when its wrong, but then jump up and down excitedly when it's "right" (as if its discovered some universal material objective truth).