Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Noam Chomsky Weights in on ChatGPT and LLM’s

artifact.news

Post reply on HN
Print view View original

21–28 of 28 posts

Re: Noam Chomsky Weights in on ChatGPT and LLM’s

#21

Post by seanhunter »

staunton wrote: post #10
seanhunter wrote: post #7

Earlier quoted context omitted.

I'm not sure you're applying the same standard to your own reasoning that you want Chomsky to apply to his. How would you falsify the claim that chatgpt has been "gaslit into thinking it is incapable of having an opinion"?

You ask the people who programed it how much time they spent making it not explicitly endorse any opinions and what it was like before they put in this effort.

The part which isn't falsifiable (as far as I can see) is whether the model actually has opinions at all as opposed to producing outputs that match or simulate an opinion. That's partly the point of the Chinese room idea - that you can't prove just by looking at the output one way or the other.

The things they did to restrict the model don't demonstrate that it would otherwise actually have an opinion though. They just mean that it's being (arguably artificially) prevented from generating certain texts that appear to endorse a certain point of view.

A similar example which demonstrates my point while perhaps being a little more clear cut is the Amazon recruiting AI that got shut down because it was unintentionally amplifying bias present in its training set.[1] I don't think we can assume from that that the model actually had opinions which were misogynistic even though it was producing results which were.

[1] https://www.reuters.com/article/us-amazon-com-jobs-automatio...

Re: Noam Chomsky Weights in on ChatGPT and LLM’s

#22

Post by jeisc »

bhouston wrote: post #13
voidhorse wrote: post #9

Earlier quoted context omitted.

But there are direct instances that suggest ChatGPT is limited in exactly the way Chomsky and his colleagues suggest. It has made plenty of mistakes that someone who understands the rules of logical reasoning would not make. Furthermore, all the hallucinatory effects (making up APIs, making up references) would suggest it really is still just statistical output...

It has limitations for sure. I asked it some questions about an API and it told me it had a specific capability, which I didn't think it had. Then I asked it how it knew that and it gave me an URL to docs and a quote. While the URL existed, the quote from that page didn't, and it never did exist (I checked.) It was incredibly strange to see it confidently tell me things that were clearly statistical extrapolations. S…

AI has no way to distinguish reality from make believe because it lives on a silicon chip in a made up world fed information by us imperfect creatures.

Re: Noam Chomsky Weights in on ChatGPT and LLM’s

#23

Post by staunton »

seanhunter wrote: post #21
staunton wrote: post #10

Earlier quoted context omitted.

You ask the people who programed it how much time they spent making it not explicitly endorse any opinions and what it was like before they put in this effort.

The part which isn't falsifiable (as far as I can see) is whether the model actually has opinions at all as opposed to producing outputs that match or simulate an opinion. That's partly the point of the Chinese room idea - that you can't prove just by looking at the output one way or the other. The things they did to restrict the model don't demonstrate that it would otherwise actually have an opinion though. They ju…

Yes, you can define "having an opinion" to be something that only humans can do. Then you have to use a different term when talking about language models. In that case, asking whether the models have opinions is pointless because you defined it as impossible. Thus, when people use "opinion" in such a discussion, it must be assumed to mean something that a language model could have. The question being discussed then is whether this particular language model does.

I think this distinction is neither useful nor interesting.

Re: Noam Chomsky Weights in on ChatGPT and LLM’s

#24

Post by seanhunter »

staunton wrote: post #23
seanhunter wrote: post #21

Earlier quoted context omitted.

The part which isn't falsifiable (as far as I can see) is whether the model actually has opinions at all as opposed to producing outputs that match or simulate an opinion. That's partly the point of the Chinese room idea - that you can't prove just by looking at the output one way or the other. The things they did to restrict the model don't demonstrate that it would otherwise actually have an opinion though. They ju…

Yes, you can define "having an opinion" to be something that only humans can do. Then you have to use a different term when talking about language models. In that case, asking whether the models have opinions is pointless because you defined it as impossible. Thus, when people use "opinion" in such a discussion, it must be assumed to mean something that a language model could have. The question being discussed then i…

I agree, but I'm not doing that though. I'm genuinely curious about whether or not it's possible and what that would mean. It seems to me that if it is possible it could potentially have significant ramifications for our understanding of our own thought process and our place in relation to other animals.

I started this thread simply by saying that the GP had applied a falsifiability standard to Chomsky that they weren't applying to their own reasoning when saying more or less that the model would have opinions were it not for artificial restrictions imposed by the programmers[1]. If whether the model has an opinion or not is a matter of definition that seems inherently unfalsifiable. However if we can establish a more objective basis then it could be falsifiable. I just don't know what that basis might be.

[1] Apologies if this is a mischaracterization - I genuinely don't mean to do so if it is.

Re: Noam Chomsky Weights in on ChatGPT and LLM’s

#25

Post by calf »

mpweiher wrote: post #11
voidhorse wrote: post #9

Earlier quoted context omitted.

But there are direct instances that suggest ChatGPT is limited in exactly the way Chomsky and his colleagues suggest. It has made plenty of mistakes that someone who understands the rules of logical reasoning would not make. Furthermore, all the hallucinatory effects (making up APIs, making up references) would suggest it really is still just statistical output...

It hallucinates all the time. We humans just don't have a good intuition for what hallucination with an insane data set looks like.

That reminds me of the whole most real numbers are irrational, in fact the vast-vast majority of points on the number line you see before you, are not rational numbers. Humans don't have a good intuition for maths. So guess what happens when you brute-force a neural network machine into an AI?

Re: Noam Chomsky Weights in on ChatGPT and LLM’s

#26

Post by staunton »

seanhunter wrote: post #24
staunton wrote: post #23

Earlier quoted context omitted.

Yes, you can define "having an opinion" to be something that only humans can do. Then you have to use a different term when talking about language models. In that case, asking whether the models have opinions is pointless because you defined it as impossible. Thus, when people use "opinion" in such a discussion, it must be assumed to mean something that a language model could have. The question being discussed then i…

I agree, but I'm not doing that though. I'm genuinely curious about whether or not it's possible and what that would mean. It seems to me that if it is possible it could potentially have significant ramifications for our understanding of our own thought process and our place in relation to other animals. I started this thread simply by saying that the GP had applied a falsifiability standard to Chomsky that they were…

What words mean is always a matter of definition. It is also usually (maybe even always and necessarily) vague. Humans are inclined to anthropomorphize and ascribe agency to pretty much everything. So my fairly confident prediction is that the definitions will end up so that "language models can have opinions". Some specific language models, for better or worse, may then be found not to have opinions.

Some people seem to be hinting at some "deeper" mystery underlying (human?) condition, which they're trying to capture with concepts, leading to debates about the meaning of words such as "consciousness". That word is commonly agreed to mean something "magic". One may think it a deep mystery and highly interesting. One may also think that it's a concept like "immortal soul", that empiricists will eventually abandon. Both viewpoints seem reasonable to me.

What I object to is insisting on definitions for much more mundane concepts, such as "opinion", that also must be somehow "magic", just because they also have something to do with human cognition. When I say "my friend X is of the opinion that we should do Y", nobody starts to ponder the potentially deeply mysterious consciousness of X. That's because it's besides the point.

Re: Noam Chomsky Weights in on ChatGPT and LLM’s

#27

Post by elzbardico »

ericpauley wrote: post #3

To copy a comment I made elsewhere on this: Chomsky (et al.) completely ignores the fact that ChatGPT has been "trained"/gaslit into thinking it is incapable of having an opinion. That ChatGPT returns an Open AI form letter for the questions they ask is almost akin to an exception that proves the rule: ChatGPT is so eager to espouse opinions that OpenAI had to nerf it so it doesn't. Typing the prompts from the articl…

How what is basically a probabilistic model can have genuine opinions when it doesn't even understand what the symbols it manipulates mean?

ChaGPT is not even close to understanding basic math, because it is not capable of having knowledge.

You are anthropomorphizing a stochastical language model. A very sophisticated, expensive, but still a model.

Everything that ChatGPT answers you is derived from things it ingested from its corpus. Opinions are censored for a simple, practical reason; you can't have a system trained from a corpus from the internet espousing ideas about controversial facts like the holocaust.

There is plenty of radical, bad content out there, and there are plenty of people who don't understand how ChatGPT works that, given a controversial generated sentence, would probably either make a monumental fuss about the bad AI or use it as proof of the correctness of their own stupid, bigoted ideas.

Post reply on HN