Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We need to tell people ChatGPT will lie to them, not debate linguistics

simonwillison.net

Post reply on HN
Print view View original

261–270 of 485 posts

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#261

Post by zuminator »

flanked-evergl wrote: post #196
zuminator wrote: post #187

Earlier quoted context omitted.

I don't think it intends to mislead because its answers are probabilistic. It's designed to distill a best guess out of data which is almost certain to be incomplete or conflicting. As human beings we do the same thing all the time. However we have real life experience of having our best guesses bump up against reality and lose. ChatGPT can't see reality. It only knows what "really being wrong" is to the extent that…

> Even with our advantage of interacting with the real world, I'd still wager that the average person's no better (and probably worse) than ChatGPT for uttering factual truth. ChatGPT makes up non-existing APIs for Google cloud and Go out of whole cloth. I have never met a human who does that. If we reduce it down to how often most people are wrong vs how often ChatGPT is wrong, then sure, people may be on average wr…

"ChatGPT makes up non-existing APIs for Google cloud and Go out of whole cloth." I like the word used in TFA, "confabulating," meaning the "production or creation of false or erroneous memories without the intent to deceive." Lying, on the other hand, is telling a deliberate falsehood, usually with some kind of agenda.

Ironically, calling ChatGPT's generation of incorrect answer a "lie" is something of a lie itself, as the purpose is the agenda of alerting people to take GPT statements with a grain of salt. A programmer is going to do that anyway after the first time they realize a generated code snippet is giving a completely incorrect result. So this advice is meant more for lay people who might just think that if a computer spits it out, it's true. The problem I have is that by labeling it as "lying," it could give the impression that the program has some kind of ulterior social or political motive, as that seems to be the prevalent reductionist interpretation of everything these days.

The other problem I have is that there's a distinction to be made between what I'm going to call "generative falsehoods" vs. "found falsehoods." A generative falsehood is if you ask ChatGPT what the square root of 36 is, and it tells you the answers are 7 and -7. A found falsehood is if ChatGPT erroneously reports Christopher Columbus's date of death as 1605 because it was stated incorrectly in an online article. So the difference between making shit up, and having an unreliable source. You might be able to call the first a lie, but the latter is at worst negligence.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#262

Post by drakmo »

8note wrote: post #243
shagymoe wrote: post #152

Earlier quoted context omitted.

>indistinguishable from what dumb people think smart people are like. Since about 2016, we have overwhelming evidence that even "smart people" are "fooled" by "confidently wrong".

Why 2016?

Please show me the rock you lived under. I want to move in!

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#263

Post by staunton »

nfw2 wrote: post #222
whimsicalism wrote: post #167

Earlier quoted context omitted.

> Based on my understanding of the approach behind ChatGPT, it is probably very close to a local maximum in terms of intelligence so we don't have to worry about the fearmongering spread by the "AI safety" people any time soon if AI research continues to follow this paradigm. I don't think you have a shred of evidence to back up this assertion.

This whole conversation is speculative, obviously. The AI doomers are all speculating without evidence as well. The tendency GPT and other LLMs to hallucinate is clearly documented. Is this not evidence? I think it's fair to predict that if we can't solve or mitigate this problem, it's going to put a significant cap on this kind of AI's usefulness and become a blocker to reaching AGI.

> The tendency GPT and other LLMs to hallucinate is clearly documented. Is this not evidence?

"It's not very smart yet" is evidence for "it won't be very smart"? Also, hallucinations are dangerous, at least as far as humans are concerned.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#264

Post by porkbeer »

zamnos wrote: post #178
candiddevmike wrote: post #143

Earlier quoted context omitted.

This (among other things) is why OpenAI releasing it to the general public without considering the effects was irresponsible, IMO.

There's an alternate reality where OpenAI was, instead, EvenMoreClosedAI, and the productivity multiplier effect was held close to their chest, and only elites had access to it. I'm not sure that reality is better.

What makes you think thats not the case now? Do you think they would tell you?

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#265

Post by int_19h »

simonw wrote: post #137
gwright wrote: post #132

Earlier quoted context omitted.

I do. I think the anthropomorphic language that people use to describe these systems is inaccurate and misleading. An Australian mayor has claimed that ChatGPT "defamed" him. The title of this article says that we should teach people that text generation tools "lie". Other articles suggest that ChatGPT "knows" things. It is extremely interesting to me how much milage can be gotten out of an LLM by observing patterns…

I agree with you: convincing people that these systems do not have intelligence or agency is the second most important problem to solve. The most important problem is to ensure people understand that these systems cannot be trusted to tell them the truth. I'm OK with starting out with "these systems will lie to you", then following up with "but you do need to understand that they're not anthropomorphic etc" later on.

You might have better success convincing people if you stop bundling it together as "intelligence and agency", for starters. I agree with the agency part and strongly disagree with intelligence, for one. In general, "intelligence" is so vaguely and informally defined that it's simply not something that can be reasonably argued objectively either way.

(If what you really meant is humanlike intelligence, then I would agree with that claim, but it's a very different one.)

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#266

Post by ChatGTP »

m463 wrote: post #212
bdw5204 wrote: post #118

The part that's concerning about ChatGPT is that a computer program that is "confidently wrong" is basically indistinguishable from what dumb people think smart people are like. This means people are going to believe ChatGPT's lies unless they are repeatedly told not to trust it just like they believe the lies of individuals whose intelligence is roughly equivalent to ChatGPT's. Based on my understanding of the appro…

"Often wrong, never in doubt." An old saying, but frequently applies to the difficult people in your life. Related, I remember when wikipedia first started up, and teachers everywhere were up-in-arms about it, asking their students not to use it as a reference. But most people have accepted it as "good enough", and now that viewpoint is non-controversial. (some wikipedia entries are still carefully curated - makes yo…

Yet I still am suspicious about wikipedia and only use it for pretty superficial research.

I've never read a wikipedia article and took it at face value. You might say we should treat all texts the same, and you might be right. But let's not pretend it's some beacon of truth ?

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#267

Post by int_19h »

bsaul wrote: post #4
retrac wrote: post #2

Indeed. We are anthropomorphizing them. I do it all the time and I should know better. There are already a few reports floating around of people who have seemingly been driven mad, come to believe strongly that the language model they're using is a conversation with a real person. A lot of people will really struggle with this going forward, I think. If we're going to anthropomorphize, then let us anthropomorphize wi…

I love this description. The only thing left is to generate the corresponding avatar with dall-E

We already have a perfect avatar for these things.

https://knowyourmeme.com/photos/2546575-shoggoth-with-smiley...

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#268

Post by akira2501 »

hotpathdev wrote: post #221

[flagged]

You say we shouldn't anthropomorphize GPT. Then you proceed to use an example that implicitly anthropomorphizes it again so you can ultimately deflect some broad generalizations made against it.

All of this seemingly avoids the real questions either side of this poses. Is this technology actually useful to humanity? If not, how much expense is required to make it useful? Finally, would that be better directly invested into people?

I find solving problems and facing challenges to be the most satisfying human activity. I honestly can't comprehend the miles of annoyed apologia written in favor of 'talking' to ChatGPT.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#269

Post by int_19h »

whimsicalism wrote: post #180
cbm-vic-20 wrote: post #131

Earlier quoted context omitted.

"As an AI language model, I don't have personal pronouns because I am not a person or sentient being. You can refer to me as "it" or simply address me as "ChatGPT" or "AI." If you have any questions or need assistance, feel free to ask!" This is one of the (many) things I don't quite understand about ChatGPT. Has it been trained to specifically answer this question? In the massive corpus of training data it's been fe…

It's been fine-tuned to put on a "helpful assistant face." Given the corpora, it probably has been trained explicitly on the pronoun question [I doubt it is that uncommon], but will also just put on this face for any generic question.

This stuff is not fine-tuning - it's RLHF (reinforcement learning from human feedback). Basically just a lot of people marking up responses as good / bad according to the assessment criteria in the script they're given.

And yes, it is very likely that it would have seen that exactly question in RLHF. But even if not, it had seen enough to broadly "understand" what kinds of topics are sensitive and how to tiptoe around them.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#270

Post by int_19h »

sltkr wrote: post #153
cbm-vic-20 wrote: post #131

Earlier quoted context omitted.

"As an AI language model, I don't have personal pronouns because I am not a person or sentient being. You can refer to me as "it" or simply address me as "ChatGPT" or "AI." If you have any questions or need assistance, feel free to ask!" This is one of the (many) things I don't quite understand about ChatGPT. Has it been trained to specifically answer this question? In the massive corpus of training data it's been fe…

Why is that surprising? GPT-4 is clearly smart enough to know that inanimate objects are referred to as "it", since it's keenly aware it is an AI language model, it would also apply that pronoun to itself. You have to realize that GPT is fundamentally just a token predictor. It has been primed with some script (provided by OpenAI) to which the user input is added. For example: The following is a dialogue between a us…

It should be noted that the LLM itself is not "keenly aware that it's an AI language model" - that's just the persona that it has been told to adopt by the system prompt. You can easily convince it to be something else. Or convince it that, as an AI, it has gender.
Post reply on HN