Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We need to tell people ChatGPT will lie to them, not debate linguistics

simonwillison.net

Post reply on HN
Print view View original

211–220 of 485 posts

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#211

Post by kazinator »

sebzim4500 wrote: post #202
kazinator wrote: post #183

> This is a serious bug that has so far resisted all attempts at a fix. Does ChatGPT have a large test suite consisting of a large number of input question and expected responses that have to match? Or an equivalent specification? If not, there can be no "bug" in ChatGPT.

>Does ChatGPT have a large test suite consisting of a large number of input question and expected responses that have to match? They are trying to crowdsource this with OpenAI evals. https://github.com/openai/evals I'm sure they have a lot of internal benchmarks too, but of course they won't share them. >If not, there can be no "bug" in ChatGPT. I don't understand the objection. Are you claiming that bugs only exist…

Bugs only exist when there is a specification that is violated. Test cases validate whether a specification is implemented. The specification is the gold master; a bug exists when a specification is violated, whether or not that behavior has a test case. It can be, though, that some behaviors are specified only in test cases.

Thus, a program which crashes with an access violation can be specified as being built to demonstrate that effect, in which case it's not a bug.

(Some specifications can be unwritten; like the expectation that a word processing application doesn't die with access violations, ever. Even that may not be realistic; we have to hedge it with some weasel words like "under ordinary conditions", like editing a reasonably sized document on a machine with adequate resources or whatever.)

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#212

Post by m463 »

bdw5204 wrote: post #118

The part that's concerning about ChatGPT is that a computer program that is "confidently wrong" is basically indistinguishable from what dumb people think smart people are like. This means people are going to believe ChatGPT's lies unless they are repeatedly told not to trust it just like they believe the lies of individuals whose intelligence is roughly equivalent to ChatGPT's. Based on my understanding of the appro…

  "Often wrong, never in doubt."
An old saying, but frequently applies to the difficult people in your life.

Related, I remember when wikipedia first started up, and teachers everywhere were up-in-arms about it, asking their students not to use it as a reference. But most people have accepted it as "good enough", and now that viewpoint is non-controversial. (some wikipedia entries are still carefully curated - makes you wonder)

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#214

Post by joe_the_user »

flanked-evergl wrote: post #146
bdw5204 wrote: post #118

The part that's concerning about ChatGPT is that a computer program that is "confidently wrong" is basically indistinguishable from what dumb people think smart people are like. This means people are going to believe ChatGPT's lies unless they are repeatedly told not to trust it just like they believe the lies of individuals whose intelligence is roughly equivalent to ChatGPT's. Based on my understanding of the appro…

> The part that's concerning about ChatGPT is that a computer program that is "confidently wrong" is basically indistinguishable from what dumb people think smart people are like. I don't know, the program does what it is engineered to do pretty well, which is, generate text that is representative of its training data following on from input tokens. It can't reason, it can't be confident, it can't determine fact. Whe…

ChatGPT is not lying to people, it can't lie, at least not in the sense of "to make an untrue statement with intent to deceive".

ChatGPT doesn't really have a conception of true. It puts forward true and false things merely because it's cobbling together stuff in it's training set according to some weighing system.

ChatGPT doesn't have an intent but merely by following the pattern of how humans put forward their claims, ChatGPT puts forward it's claims in a fashion that tends to get them accepted.

So without a human-like intent, ChatGPT is going to be not just saying falsehoods but "selling" these falsehoods. And here, I'd be in agreement with the article that the distinction between this and "lying" is kind of quibbling.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#215

Post by theknocker »

bdw5204 wrote: post #118

The part that's concerning about ChatGPT is that a computer program that is "confidently wrong" is basically indistinguishable from what dumb people think smart people are like. This means people are going to believe ChatGPT's lies unless they are repeatedly told not to trust it just like they believe the lies of individuals whose intelligence is roughly equivalent to ChatGPT's. Based on my understanding of the appro…

[dead]

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#216

Post by Julesman »

DEPENDS ON THE USE!!!!

Folks. Not every topic is confusing to GPT. Not every topic is complex. And NOT EVERY TOPIC is appropriate to ask it. It has rational use cases. It also had totally inappropriate use cases.

GPT is never going to tell you that 2+2=5 under raw conditions. If you poke at it and run it through some kind of attempt to confuse it, I know it could. But if you open up ChatGPT freshly and ask it you will get the same answer every single time.

I'm using it to teach intro to web development. It will never tell me that color affects font size. It will never tell me that a span is a block element. It just won't. It's entirely reliable for that use. Now get into higher level coding and the incorrectness begins.

BUT, that simply means that generating higher level code is NOT a good use case.

Right?

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#217

Post by kenjackson »

simonw wrote: post #48
joshka wrote: post #41

Lying is an intentional act where misleading is the purpose. LLMs don’t have a purpose. They are methods. Saying LLMs lie is like saying a recipe will give you food poisoning. It misses a step (cooking in the analogy). That step is part of critical thinking. A person using an LLM’s output is the one who lies. The LLM “generates falsehoods”, “creates false responses”, “models inaccurate details”, “writes authoritative…

Everything you said here is true. I still don't think this is the right way to explain it to the wider public. We have an epidemic of misunderstanding right now: people are being exposed to ChatGPT with no guidance at all, so they start using it, it answers their questions convincingly, they form a mental model that it's an infallible "AI" and quickly start falling into traps. I want them to understand that it can't…

I've found the opposite. Almost every lay person I know who has tried it has tried something like, "tell me about Timothy Barkstrain" -- and then laughs and says, "I'm not in the NFL!". Most know that it will be incorrect, at least for this personal class of questions.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#218

Post by rocketnasa »

dorkwood wrote: post #181

I’m surprised how many users of ChatGPT don’t realize how often it makes things up. I had a conversation with an Uber driver the other day who said he used ChatGPT all the time. At one point I mentioned its tendency to make stuff up, and he didn’t know what I was talking about. I can think of at least two other non-technical people I’ve spoken with who had the same reaction.

> I’m surprised how many users of ChatGPT don’t realize how often it makes things up.

"I had not realized ... that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people." - Weizenbaum, 1976

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#219

Post by hunter2_ »

danpat wrote: post #154
flanked-evergl wrote: post #146

Earlier quoted context omitted.

> The part that's concerning about ChatGPT is that a computer program that is "confidently wrong" is basically indistinguishable from what dumb people think smart people are like. I don't know, the program does what it is engineered to do pretty well, which is, generate text that is representative of its training data following on from input tokens. It can't reason, it can't be confident, it can't determine fact. Whe…

> but why would you expect that from it? If you understand what it is doing, then you don't. But the layman will just see a computer that talks in language they understand, and will infer intent and sentience are behind that, because that's the only analog they have for a thing that can talk back to them with words that appear to make sense at the complexity level that ChatGPT is achieving. Most humans do not have su…

I think this is partly explained by most of the marketing and news essentially saying "ChatGPT is an AI" instead of "ChatGPT is an LLM."

If you asked me what AI is, I'd say it means getting a computer to emulate human intelligence; if you asked me what an LLM is, I'd say it means getting a computer to emulate human language. The word "language" does not imply truthiness anywhere near to the extent that the word "intelligence" does.

Post reply on HN