Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Emily Bender meant by "stochastic parrots"

spectrum.ieee.org

Post reply on HN
Print view View original

251–260 of 280 posts

Re: What Emily Bender meant by "stochastic parrots"

#251

Post by ToValueFunfetti »

dwa3592 wrote: post #245
ToValueFunfetti wrote: post #229

Earlier quoted context omitted.

If someone substituted all of your sensory inputs for something else for your entire life, how would you notice? If you wore contacts from birth that made the sky red and earbuds that censored when people said it was blue, on what basis would you realize that was wrong? I don't see what this says about the architecture of your brain, and I don't think it's the point being made in the paper. That the training data mus…

>>If someone substituted all of your sensory inputs for something else for your entire life, how would you notice? I don't know how or if I will notice. That's the biology, chemistry and physics of the brain that I don't know. I hope someone is looking into it. But this does not mean in any way that LLMs are similar to our brains!!!!!! WE DON'T KNOW HOW OUR BRAINS WORK. So going back to language modeling - language m…

I'm quite sure that LLMs are not one-to-one with our brains. It would at least be surprising if instrumental convergence was that powerful, and, even in that case, they're still not getting hungry or sleepy or having dreams, etc. At best they'd match up with a smaller region of our brains, and I'd take 9:1 odds against a close match.

But the paper argues a much stronger point than "LLMs are not similar to human brains": it claims LLMs cannot engage with meaning because they model human language instead of sensory input or human thought.

I also don't think human brains are particularly deterministic.

Re: What Emily Bender meant by "stochastic parrots"

#252

Post by naasking »

PaulDavisThe1st wrote: post #173
naasking wrote: post #156

Earlier quoted context omitted.

I think you need understanding to reason, but you don't need reasoning to understand. A child understands how to catch a ball without reasoning about forces, air resistance, gravity, etc. I think LLMs understand without reasoning. They've built a large associative network of concepts (a kind of understanding), but we don't yet have a good handle on the process of reasoning using that network.

I don't think it is useful to say that a child "understands how to catch a ball", even though it is something many of us do say quite often. The child knows how to catch the ball, without understanding. Later, the child learns both reason and physics, and can reason about ball catching in a different way. I don't think that it is useful to say that LLMs understand anything they say, or that we say to them.

> The child knows how to catch the ball, without understanding.

I disagree that decoupling "knowing" from "understanding" is well motivated or useful in any way.

> I don't think that it is useful to say that LLMs understand anything they say, or that we say to them.

This is starting to seem like a Chinese Room disagreement, so I'll just say that what that thought experiment demonstrated is that there is no robust or meaningful way to distinguish semantics from syntactic processing given everything we know about reality, and that LLMs were a fantastic demonstration of this illusion, eg. that "semantics" are reducible to a relational network of syntactic elements.

Re: What Emily Bender meant by "stochastic parrots"

#253

Post by delis-thumbs-7e »

andrewla wrote: post #242
delis-thumbs-7e wrote: post #213

Earlier quoted context omitted.

I am no linguist, but I believe this is referred to as surface structure and deep structure. What you describe is a line of text that is grammatically somewhat adequate to pass as readable and you treat all text the same. when we read the text, we decipher meaning out of according to word references and syntax - just like with Python or C++. However, if this were solely the case, we could not read Finnegan’s Wake. Pr…

What I'm saying is that we cannot observe deep structure. All we can observe is the surface structure. If we have access to the author or speaker we can ask clarifying questions, etc., but even that just produces more examples of surface structure. But even here, in the Chomsky sense, LLMs clearly exhibit deep structure because they can write at length in an internally consistent manner. Importantly, early generation…

I disagree on your definition of deep structure. We can indeed deep structure, that is my whole point, without it Finnegan’s Wake would be impossible to read. This really is something LLM’s cannot do, at least for now. And if you look at the paper I linked, if they had a world model they would not get confused on mathematics when we change the values of variables. So there seems to be plenty of things LLM’s cannot do.

And basically the burden of proof is on your side. As often is the case, discussions on AI start to resemble discussions centred on religion and belief in god. Your only line left here is “yes but you cannot prove LLM’s are conscious” - which I can’t, since proving that Unicorns do not exist is pretty senseless activity.

So basically I’d say LLM’s are stochastic parrots (or whatever term you want to use) until we can very definitely prove that they indeed think and have some sort of consciousness.

Re: What Emily Bender meant by "stochastic parrots"

#254

Post by nyeah »

dTal wrote: post #247
nyeah wrote: post #197

Earlier quoted context omitted.

No, they were correct. In fact an LLM stitches together stuff it observed in its training data. That scales up way better than a lot of us expected, but it's still correct. If you train it on lots of working code, then it's useful for coding. If you trained it primarily on non-working code it would produce nonsense.

They do not "stitch together" anything. Neither on a technical level, nor a philosophical one. It "scales better than you expected" because your mental model is wrong . And, not to insult you, but it's quite obviously wrong. As a mental model it fails to explain basic capabilities. How can an LLM follow elaborate instructions? How can it respond appropriately to user input, when the user input doesn't match any previ…

Balancing parentheses requires semantic understanding?

Look, some folks are more impassioned about this stuff than I am. Maybe that's a good thing. But LLMs do in fact just try to predict the next token, using a very big training set. They're very impressive (at tasks the training set prepares them for). But that's how they work.

Re: What Emily Bender meant by "stochastic parrots"

#255

Post by throwaway7356 »

fluoridation wrote: post #232
ToValueFunfetti wrote: post #231

Earlier quoted context omitted.

A bird doesn't learn gravity or aerodynamics, it has no 'sense of physics'. It has sensory neural activity that a scientist can show is tied to these things, but you could, at least in theory, falsify the entire experience of the bird. There is nothing in a bird's brain that directly percieves reality. Colors and sounds and textures and so on are all false primitives that don't exist in nature without us, if that's c…

>A bird doesn't learn gravity or aerodynamics, it has no 'sense of physics'. That's not what I said. What I said was that it's physics that provides the ground truth. >you could, at least in theory, falsify the entire experience of the bird It wouldn't be a bird anymore, but a dysfunctional cyborg with false perceptions. >There is nothing in a bird's brain that directly percieves reality. Yes, of course there is. Ani…

> nor do they provide the brain an interpretation of what they perceive

> What it gets from the body is raw physical measurements

No, sensory organs like eyes do a lot of processing ("interpreation"). They certainly don't send "raw physical measurements" to the brain.

Re: What Emily Bender meant by "stochastic parrots"

#256

Post by mike_hearn »

nyeah wrote: post #254
dTal wrote: post #247

Earlier quoted context omitted.

They do not "stitch together" anything. Neither on a technical level, nor a philosophical one. It "scales better than you expected" because your mental model is wrong . And, not to insult you, but it's quite obviously wrong. As a mental model it fails to explain basic capabilities. How can an LLM follow elaborate instructions? How can it respond appropriately to user input, when the user input doesn't match any previ…

Balancing parentheses requires semantic understanding? Look, some folks are more impassioned about this stuff than I am. Maybe that's a good thing. But LLMs do in fact just try to predict the next token, using a very big training set. They're very impressive (at tasks the training set prepares them for). But that's how they work.

It requires some level of semantic understanding, like what a paren is and what it means to balance them.

The issue in this discussion is that "predict the next token" is a problematically reductive description of what's going on. It's like saying compilers are programs that emit bytes or that humans are mammals that make sounds. It's not strictly false but it's not capturing the depth of what's happening either.

A simple way to see this is to ask: predicting the next token of what? The obvious answer - predicting the next token that would be found in the training set - isn't correct. If that's what it were doing then it would yield no prediction or random predictions for any prefix not found in that training set, but it isn't what happens. We see generalization and reasoning. They can answer questions never asked before. And once post-training kicks in the question of what it's predicting becomes even harder. It becomes more like predicting what this specific AI assistant would say next, which is a circular definition.

Re: What Emily Bender meant by "stochastic parrots"

#257

Post by supern0va »

throwaway7356 wrote: post #255
fluoridation wrote: post #232

Earlier quoted context omitted.

>A bird doesn't learn gravity or aerodynamics, it has no 'sense of physics'. That's not what I said. What I said was that it's physics that provides the ground truth. >you could, at least in theory, falsify the entire experience of the bird It wouldn't be a bird anymore, but a dysfunctional cyborg with false perceptions. >There is nothing in a bird's brain that directly percieves reality. Yes, of course there is. Ani…

> nor do they provide the brain an interpretation of what they perceive > What it gets from the body is raw physical measurements No, sensory organs like eyes do a lot of processing ("interpreation"). They certainly don't send "raw physical measurements" to the brain.

And even then, it's just sampling. Much of what we "see" is a prediction, and there are plenty of optical illusions out there premised on that (plus VR techniques like foveated rendering that take advantage).

Re: What Emily Bender meant by "stochastic parrots"

#258

Post by supern0va »

fluoridation wrote: post #214
ToValueFunfetti wrote: post #210

Earlier quoted context omitted.

Fair point, and on its own it would be surprising to learn what "five" means from that sentence. But you can extrapolate- across a billion sentences, there will be "the next sentence has five words"s and "this sentence are grammared wrong" and so on. It would not be at all impossible to ground a world model on pure text for that reason. And 'not impossible' is sufficient to invalidate the paper's argument.

Let me provide a less superficial response, then. >If multimodal models were still stochastic parrots by the original argument, humans would have to be as well; we don't have any way to ground anything beneath sense data Animals don't passively learn from their perceptions, don't have a separation between training and inference, and don't have a prompt-response execution model. Besides its fundamental biology, the gr…

>and even if it requests another angle and is given it, it lacks the capacity to learn that new information permanently.

I'd argue this isn't true today, but that the loop for incorporation is long (ie, the next training or finetuning run).

>A multimodal model knows about images of pipes and facts about pipes, but doesn't know pipes; it doesn't have literally first-hand experience with them.

Wouldn't this mean that any human who hasn't seen a pipe in person or interacted with it, similarly doesn't "know" a pipe? Most of us haven't interacted with the vast majority of "things" in the world, yet we're still able to build a model and abstractions for them such that we can reason about them, right?

Re: What Emily Bender meant by "stochastic parrots"

#259

Post by [deleted] »

mike_hearn wrote: post #256
nyeah wrote: post #254

Earlier quoted context omitted.

Balancing parentheses requires semantic understanding? Look, some folks are more impassioned about this stuff than I am. Maybe that's a good thing. But LLMs do in fact just try to predict the next token, using a very big training set. They're very impressive (at tasks the training set prepares them for). But that's how they work.

It requires some level of semantic understanding, like what a paren is and what it means to balance them. The issue in this discussion is that "predict the next token" is a problematically reductive description of what's going on. It's like saying compilers are programs that emit bytes or that humans are mammals that make sounds. It's not strictly false but it's not capturing the depth of what's happening either. A s…

[deleted]

Re: What Emily Bender meant by "stochastic parrots"

#260

Post by nyeah »

mike_hearn wrote: post #256
nyeah wrote: post #254

Earlier quoted context omitted.

Balancing parentheses requires semantic understanding? Look, some folks are more impassioned about this stuff than I am. Maybe that's a good thing. But LLMs do in fact just try to predict the next token, using a very big training set. They're very impressive (at tasks the training set prepares them for). But that's how they work.

It requires some level of semantic understanding, like what a paren is and what it means to balance them. The issue in this discussion is that "predict the next token" is a problematically reductive description of what's going on. It's like saying compilers are programs that emit bytes or that humans are mammals that make sounds. It's not strictly false but it's not capturing the depth of what's happening either. A s…

In fact LLMs are trained to predict the next token in the training set. Of course sometimes a new text input doesn't match the training set, or it matches two or more places in the training set. LLMs use a neural network to interpolate, so that's fine. Please look this up if you have any doubts.

Ok. Now. I think you're adding something to the description above. Maybe what you're describing is something "emergent," or maybe it's basically just word vectors that were built in on purpose. You may be adding something correct, or something incorrect. Fine.

But it's not reasonable to say that the "reductive" description above is a "lie". It's not. It's more like a recipe. If you look at correct instructions for making steak, and you call the author a "liar" then you are missing something important.

Post reply on HN