Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We need to tell people ChatGPT will lie to them, not debate linguistics

simonwillison.net

Post reply on HN
Print view View original

321–330 of 485 posts

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#321

Post by ramblenode »

brookst wrote: post #250
kerkeslager wrote: post #230

Earlier quoted context omitted.

> If you ask someone off the street a question, they won't know what you are talking about because their mind is going to be mid thought, but being friendly they will give you a low-effort first guess. That's lying. If you don't know the answer, the correct answer is "I don't know", or to ask clarifying questions. To pretend that you know when you don't, is to lie. Even if your intent is to deceive because you think…

> That's lying. If you don't know the answer, the correct answer is "I don't know", or to ask clarifying questions. To pretend that you know when you don't, is to lie. You can use that definition, but it is not how the word "lie" is commonly used. Mirriam-Webster defines lie as "to make an untrue statement with the intent to deceive"[0] Cambridge dictionary defines lie as "to say or write something that is not true i…

> You can use that definition, but it is not how the word "lie" is commonly used.

The problem is that any word that ascribes agency to the LLM will technically be incorrect. But that removes most possible descriptions of its tone and style which are a relevant part of its response.

An analogy would be if ChatGPT started insulting me and calling my question stupid. Would it be wrong to call its response "rude" or "mean" just because it is statistically regurgitating text that matches some input parameters? This seems unreasonable if our goal is to capture the gist of its response.

This is why people judge it to be "lying" rather than being merely incorrect: it is responding with a certain conversational tone in a certain context that gives its answer a style of arrogance, deceitfulness, and narcissism (because I guess that's what internet comment boards are filled with). "Lying" is a description of the totality of its response--including tone and style--not just the truth value of the answer.

If we are to be really pedantic, the LLM isn't even correct or incorrect ;it is just completing strings of tokens. Humans are imputing their own judgment about what those tokens mean--same as with tone and style. Imputing tone isn't so different from imputing truth value.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#322

Post by stephendause »

nfw2 wrote: post #222
whimsicalism wrote: post #167

Earlier quoted context omitted.

> Based on my understanding of the approach behind ChatGPT, it is probably very close to a local maximum in terms of intelligence so we don't have to worry about the fearmongering spread by the "AI safety" people any time soon if AI research continues to follow this paradigm. I don't think you have a shred of evidence to back up this assertion.

This whole conversation is speculative, obviously. The AI doomers are all speculating without evidence as well. The tendency GPT and other LLMs to hallucinate is clearly documented. Is this not evidence? I think it's fair to predict that if we can't solve or mitigate this problem, it's going to put a significant cap on this kind of AI's usefulness and become a blocker to reaching AGI.

I believe the relevant assertion is

> Based on my understanding of the approach behind ChatGPT, it is probably very close to a local maximum in terms of intelligence

To me that is an open question. OpenAI has not revealed a lot of the relevant stats behind GPT-4. I also haven't seen anything about their future LLM research (though I haven't looked very hard), but I think the above assertion remains to be seen.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#323

Post by [deleted] »

nfw2 wrote: post #222
whimsicalism wrote: post #167

Earlier quoted context omitted.

> Based on my understanding of the approach behind ChatGPT, it is probably very close to a local maximum in terms of intelligence so we don't have to worry about the fearmongering spread by the "AI safety" people any time soon if AI research continues to follow this paradigm. I don't think you have a shred of evidence to back up this assertion.

This whole conversation is speculative, obviously. The AI doomers are all speculating without evidence as well. The tendency GPT and other LLMs to hallucinate is clearly documented. Is this not evidence? I think it's fair to predict that if we can't solve or mitigate this problem, it's going to put a significant cap on this kind of AI's usefulness and become a blocker to reaching AGI.

[deleted]

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#324

Post by afro88 »

> ChatGPT will lie to you

>

> Or

>

> ChatGPT doesn’t lie, lying is too human and implies intent. It hallucinates. Actually no, hallucination still implies human-like thought. It confabulates. That’s a term used in psychiatry to describe when someone replaces a gap in one’s memory by a falsification that one believes to be true—though of course these things don’t have human minds so even confabulation is unnecessarily anthropomorphic. I hope you’ve enjoyed this linguistic detour!

Classic strawman. Third option: ChatGPT gets things wrong. There you go, problem solved.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#325

Post by gwright »

IIAOPSW wrote: post #294
gwright wrote: post #132

Earlier quoted context omitted.

I do. I think the anthropomorphic language that people use to describe these systems is inaccurate and misleading. An Australian mayor has claimed that ChatGPT "defamed" him. The title of this article says that we should teach people that text generation tools "lie". Other articles suggest that ChatGPT "knows" things. It is extremely interesting to me how much milage can be gotten out of an LLM by observing patterns…

I disagree with your confident assertions about its "agency" and "intent" when there's no goal post for what those things mean in humans to begin with. Barring OpenAI's filters, if I asked it to participate as a party in a business negotiation determining a fair selling price for its writing services, I'm sure it could emulate that sort of character long enough to pass. And if it can successfully fake being a self in…

> I disagree with your confident assertions about its "agency" and "intent" when there's no goal post for what those things mean in humans to begin with.

I'm not sure how confident I am!

I think the success of LLMs are serving to challenge our understandings of cognition and intelligence in humans. I tend to agree with you that our understanding of agency, intent, intelligence, cognition, free will, and related ideas in humans is incomplete and so it is a challenge to think about how they might apply (or not) to LLMs.

> whats the difference between faking it and actually having agency?

Good question. The Chinese Room Argument is very close to this but framed as "understanding" vs "agency": https://plato.stanford.edu/entries/chinese-room/

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#326

Post by antupis »

dwaltrip wrote: post #312
nfw2 wrote: post #222

Earlier quoted context omitted.

This whole conversation is speculative, obviously. The AI doomers are all speculating without evidence as well. The tendency GPT and other LLMs to hallucinate is clearly documented. Is this not evidence? I think it's fair to predict that if we can't solve or mitigate this problem, it's going to put a significant cap on this kind of AI's usefulness and become a blocker to reaching AGI.

Gpt-4 hallucinates meaningfully less than gpt-3. There is more evidence in favor of “more improvements to come” than “ai winter approaches”. A lot smart people seem to be saying that the existing approaches have room to improve simply by training with more data. Based on what I’ve read and roughly speculating, it looks there is easily enough existing data for gpt-5 and probably a few more versions after. I’m not sure…

I think good annotation is much harder than obtaining data. But now Reddit, Twitter and Quora will realise what kind goldmine their data is they might close easy access to it.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#327

Post by barbazoo »

HybridCurve wrote: post #318
pessimizer wrote: post #295

Earlier quoted context omitted.

The Matrix turned out worse:)

If you just look at the jihad, maybe it was comparable (As Frank eluded to it). But then there was Muad'dib's subsequent campaign against the known universe which killed billions, Leto II's golden path spanning 3,500 years of tyranny and the famine times before scattering. If I had to choose, it would be the blue pill, though chairdogs do contribute to a strong counter-argument.

Let me guess, Star Trek?

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#328

Post by berniedurfee »

I had an interesting conversation with ChatGPT yesterday. I asked what the heredoc operator was in Terraform that removes white space.

It replied that I should use I tried it and got an error. A bit of Googling and I found the correct operator is I told ChatGPT and it apologized and said I was indeed right. Then it proceeded to give me another example, again with the incorrect operator as The next example used Another couple attempts and it started to respond correctly with the I was really surprised that it learned so quickly on the fly.

I hadn’t thought of the implications until I read this article. Can LLMs be gaslighted in realtime?

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#329

Post by fnordpiglet »

berniedurfee wrote: post #328

I had an interesting conversation with ChatGPT yesterday. I asked what the heredoc operator was in Terraform that removes white space. It replied that I should use I tried it and got an error. A bit of Googling and I found the correct operator is I told ChatGPT and it apologized and said I was indeed right. Then it proceeded to give me another example, again with the incorrect operator as The next example used Anothe…

To my understanding the “chat” part means the entire context of the conversation becomes a conditional to the next prompt (forgive me if I used the wrong term), so in a sense it can be trained in the convo. You can definitely gas light it, including building up a context that subverts it’s guard rails. A lot of focus is on the one shot subversion but you can achieve it too more subtly.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#330

Post by ramblenode »

flanked-evergl wrote: post #146
bdw5204 wrote: post #118

The part that's concerning about ChatGPT is that a computer program that is "confidently wrong" is basically indistinguishable from what dumb people think smart people are like. This means people are going to believe ChatGPT's lies unless they are repeatedly told not to trust it just like they believe the lies of individuals whose intelligence is roughly equivalent to ChatGPT's. Based on my understanding of the appro…

> The part that's concerning about ChatGPT is that a computer program that is "confidently wrong" is basically indistinguishable from what dumb people think smart people are like. I don't know, the program does what it is engineered to do pretty well, which is, generate text that is representative of its training data following on from input tokens. It can't reason, it can't be confident, it can't determine fact. Whe…

> ChatGPT is not lying to people, it can't lie

I think the discussion of whether an LLM can technically lie is a red herring.

The answer you get from an LLM isn't just a set of facts and truth values; it is also a conversational style and tone. It's training data isn't a graph of facts; it's human conversation, including arrogance, deflection, defensiveness, and deceit. If the LLM regurgitates text in a style that matches our human understanding of what a narcissistic and deceitful reply looks like, it seems reasonable we could call the response deceitful. The conversation around whether ChatGPT can technically lie seems to just be splitting hairs over whether the response is itself a lie or is merely an untrue statement in the style of a lie--a distinction which probably isn't meaningful most of the time.

Ultimately, tone, style, truth, and falsity are just qualia we humans are imputing onto a statistically arranged string of tokens. In the same way that ChatGPT can't lie it also can't be correct or incorrect, as that too is imputing some kind of meaning where there isn't any.

Post reply on HN