What Emily Bender meant by "stochastic parrots"

What I'm saying is that we cannot observe deep structure. All we can observe is the surface structure. If we have access to the author or speaker we can ask clarifying questions, etc., but even that just produces more examples of surface structure.

But even here, in the Chomsky sense, LLMs clearly exhibit deep structure because they can write at length in an internally consistent manner. Importantly, early generations, GPT-2 and even GPT-3, did not definitively have this property; roughly, an object that was green at the beginning of a paragraph might not still be green at the end of the paragraph. This was strong evidence for lack of a world model.

Current LLMs do not show this behavior. We cannot prove that LLMs have a world model, in fact, their architecture seems to rule it out, but looking at it from a linguistic standpoint, they produce language in a manner as if to reflect a world view. That is, we cannot easily falsify the statement "LLMs somehow represent a world model"; and current examples of "disproving" their world view are so convoluted that even humans do not appear to (observationally) have a world view either.

I'm not making a claim as to LLMs having genuine deep structure or consciousness or anything like that. I'm claiming that we can't rule out current or future capabilities or make structural assumptions. Yes, they generalize from their training data, but unless you can make very specific claims about the kinds of things that they _cannot_ do, I can't take this statement as particularly compelling.

Intelligence is plenty adaptive when you're just trying to outsmart the environment.

The notion that this is a "living fossil" is... pretty wild. It's genus octopus. Unless it's wildly misclassified (doubtful), it has a recent-ish common ancestor with most other common octopuses, which all have normal octopus behavior. You find yourself in need of extraordinary evidence.

>It wouldn't be a bird anymore, but a dysfunctional cyborg with false perceptions.

The criticism in the paper is of the architecture of LLMs, isn't it? The paper contends

"""Text generated by an LM is not grounded in communicative intent, any model of the world, or any model of the reader’s state of mind. [...] an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot"""

They're saying that the model cannot learn anything about reality irrespective of training data. Your point is an interesting one, but I think it's distinct. To your point though, this is unfalsifiable from the perspective of the "bird". I can't prove that I'm not a dysfunctional cyborg with false perceptions, which makes me wonder if that's a meaningful distinction.

>Animal sensory organs do not produce false information,

I happen to be in possession of some of these and I think this needs a "usually, under ordinary conditions". (Nitpicky and not critical to my point, but I liked the beginning of this sentence too much to edit it out)

>nor do they provide the brain an interpretation of what they perceive

Whereas this I'd argue is not true at all. My cones interact with wave-particle photons at particular wavelengths. I can't even conceptualize wave-particle duality (though some humans can), but "red" and "blue" are the bread and butter of my visual consciousness. These correspond to firings of my sensory neurons much more than they correspond to anything in reality. If that's not interpretation, what is it/what is interpretation?

>The word "real" is itself meaningless to it; they're both real in that they appear in its training corpus.

I expect we agree that I can show a multimodal model a real and a CGI picture and it can tell me which is which. I can take a CGI dragon to GPT 5 and say "look what I found in my backyard" and it will say "Yeah right". Are you saying this is something only possible thanks to RLHF or other modern techniques? That may be the case, unsure how to test that without access to pretraining-only models. Or would you say my experiment is faulty here and doesn't get to your underlying claim?

On the flipside, I could show some meh drawings of fairies to Arthur Conan Doyle, and he'd say "Whoa, this changes everything". I consider him to be one of the great rational minds of history, but he was unable to pass your test here. (In fairness, he was in his 60s and his senses may have dulled, though his belief in spiritualism at large dates to his prime).

Appreciate the conversation!

> I think there's very little evidence for that.

The evidence is literally every other evolved form of intelligence. Including, despite your speculation, octopuses. Do recall "the environment" also includes other intelligent agents an animal needs to compete with. Does that help you understand? A runaway process might be required for human level intelligence but clearly not in general.

Basically all octopuses are solitary and die after breeding. LPSOs are nested in one group of octopuses. You're proposing that one group of octopuses developed social behavior and multiple breeding, then had a bunch of descendants who went back to exactly the normal octopus behavior. Hopefully it's obvious why this fails Occam's razor.