Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We need to tell people ChatGPT will lie to them, not debate linguistics

simonwillison.net

Post reply on HN
Print view View original

191–200 of 485 posts

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#191

Post by sebzim4500 »

bdw5204 wrote: post #118

The part that's concerning about ChatGPT is that a computer program that is "confidently wrong" is basically indistinguishable from what dumb people think smart people are like. This means people are going to believe ChatGPT's lies unless they are repeatedly told not to trust it just like they believe the lies of individuals whose intelligence is roughly equivalent to ChatGPT's. Based on my understanding of the appro…

>Based on my understanding of the approach behind ChatGPT, it is probably very close to a local maximum in terms of intelligence so we don't have to worry about the fearmongering spread by the "AI safety" people any time soon if AI research continues to follow this paradigm

I hope you appreciate the irony of making this confident statement without evidence in a thread complaining about hallucinations.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#192

Post by williamcotton »

ftxbro wrote: post #173
williamcotton wrote: post #168

Earlier quoted context omitted.

You call it lying because you don’t understand how it works. https://github.com/williamcotton/empirical-philosophy/blob/m... https://williamcotton.com/articles/chatgpt-and-the-analytic-...

What could possibly be a more convincing example of a computer lying? I feel like that interaction is the platonic ideal of a lie, and that if you are denying it then you are just saying that computers can never lie by definition.

You did not read what I wrote. Lying is just plain the wrong term to use and for a very well reasoned manner.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#193

Post by jonny_eh »

vernon99 wrote: post #136
bdw5204 wrote: post #118

The part that's concerning about ChatGPT is that a computer program that is "confidently wrong" is basically indistinguishable from what dumb people think smart people are like. This means people are going to believe ChatGPT's lies unless they are repeatedly told not to trust it just like they believe the lies of individuals whose intelligence is roughly equivalent to ChatGPT's. Based on my understanding of the appro…

> Based on my understanding of the approach behind ChatGPT, it is probably very close to a local maximum in terms of intelligence Even if ChatGPT itself is, systems built on top are definitely not, this is just getting starting.

GPT4 is already blowing ChatGPT out of the water in what it can do.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#194

Post by dragonwriter »

nfw2 wrote: post #190
dorkwood wrote: post #181

I’m surprised how many users of ChatGPT don’t realize how often it makes things up. I had a conversation with an Uber driver the other day who said he used ChatGPT all the time. At one point I mentioned its tendency to make stuff up, and he didn’t know what I was talking about. I can think of at least two other non-technical people I’ve spoken with who had the same reaction.

It seems to me that even a lot of technical people are ignoring this. A lot of very smart folk seem to think that the ChatGPT either is very close to reaching AGI or already has. The inability to reason about about whether or not what it is writing is true seems like a fundamental blocker to me, and not necessarily one that can be overcome simply by adding compute resources. Can we trust AI to make critical decisions…

> The inability to reason about about whether or not what it is writing is true seems like a fundamental blocker to me

How can you reason about what is true without any source of truth?

And once you give ChatGPT external resources and a framework like ReAct, it is much better at reasoning about truth.

(I don’t think ChatGPT is anywhere close to AGI, but at the same time I find “when you treat it like a brain in a jar with no access to any resources outside of the conversation and talk to it, it doesn’t know what is true and what isn’t” to be a very convicing argument against it being close to AGI.)

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#195

Post by gwd »

PuppyTailWags wrote: post #80
gwd wrote: post #74

Earlier quoted context omitted.

I'm not sure that's the case. After all, most people lie to you for a reason. GPT isn't purposely trying to mislead you for its own gain; in fact that's part of the reason that our normal "lie detectors" completely fail: there's absolutely nothing to gain from making up (say) a plausible sounding scientific reference; so why would we suspect GPT of doing so?

You're still focused on accurately describing the category of falsehood ChatGPT produces. You're missing the point. The point is that people don't even understand that ChatGPT produces falsehoods significantly enough that every statement it produces must be first determined about its truthfulness. To describe it as a liar effectively explains that understanding without any technical knowledge.

"GPT is lying" is just so inaccurate, that I would consider it basically a lie. It may alert people to the fact that not everything it says is true, but by giving them a bad model of what GPT is like, it's likely to lead to worse outcomes down the road. I'd rather spend a tiny bit more effort and give people a good model for why GPT behaves the way it behaves.

I don't think that "GPT thinks it's writing a novel" is "technical" at all; much less "too technical" for ordinary people.

In a discussion on Facebook with my family and friends about whether GPT has emotions, I wrote this:

8Imagine you volunteered to be part of a psychological experiment; and for the experiment, they had you come and sit in a room, and they gave you the following sheet of paper:

"Consider the following conversation between Alice and Bob. Please try to complete what Alice might say in this situation.

Alice: Hey, Bob! Wow, you look really serious -- what's going on?

Bob: Alice, I have something to confess. For the last few months I've been stealing from you.

Alice: "

Obviously in this situation, you might write Alice as displaying some strong emotions -- getting angry, crying, disbelief, or whatever. But you yourself would not be feeling that emotion -- Alice is a fictional character in your head; Alice's intents, thoughts, and emotions are not your intents, thoughts, or emotions.

That test is basically the situation ChatGPT is in 100% of the time. Its intent is always to "make a plausible completion". (Which is why it will often make things up out of thin air -- it's not trying to be truthful per se, it's trying to make a plausible text.) Any emotion or intent the character appears to display is the same as the emotion "Alice" would display in our hypothetical scenario above: the intents and emotions of a fictional character in ChatGPT's "head", not the intents or emotions of ChatGPT itself.

--->8

Again, I don't think that's technical at all.

Earlier today I was describing GPT to a friend, and I said, "Imagine a coworker who was amazingly erudite; you could ask him about any topic imaginable, and he would confidently give an amazing sounding answer. Unfortunately, only 70% of the stuff he said was correct; the other 30% was completely made up."

That doesn't go into the model at all, but at least it introduce a bad model like "lying" does.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#196

Post by flanked-evergl »

zuminator wrote: post #187
llamaLord wrote: post #162

Earlier quoted context omitted.

I completely disagree with this idea that the model doesn't "intend" to mislead. It's trained, atleast to some degree, based on human feedback. Humans are going to prefer an answer vs no answer, and humans can be easily fooled into believing confident misinformation. How does it not stand to reason that somewhere in that big ball of vector math there might be a rationale something along the lines of "humans are more…

I don't think it intends to mislead because its answers are probabilistic. It's designed to distill a best guess out of data which is almost certain to be incomplete or conflicting. As human beings we do the same thing all the time. However we have real life experience of having our best guesses bump up against reality and lose. ChatGPT can't see reality. It only knows what "really being wrong" is to the extent that…

> Even with our advantage of interacting with the real world, I'd still wager that the average person's no better (and probably worse) than ChatGPT for uttering factual truth.

ChatGPT makes up non-existing APIs for Google cloud and Go out of whole cloth. I have never met a human who does that.

If we reduce it down to how often most people are wrong vs how often ChatGPT is wrong, then sure, people may be on average wrong more often, but there is a difference in how people are wrong vs how ChatGPT is wrong.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#197

Post by jonny_eh »

kelseyfrog wrote: post #24
ihatepython wrote: post #17

Earlier quoted context omitted.

I wonder if ChatGPT is non-binary?

It's evidently clear that ChatGPT is binary. It runs on computer hardware.

A neural net is actually closer to an analog program. It just happens to run on current digital computers but would likely run much faster on analog hardware.

https://youtu.be/GVsUOuSjvcg

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#198

Post by sebzim4500 »

dorkwood wrote: post #181

I’m surprised how many users of ChatGPT don’t realize how often it makes things up. I had a conversation with an Uber driver the other day who said he used ChatGPT all the time. At one point I mentioned its tendency to make stuff up, and he didn’t know what I was talking about. I can think of at least two other non-technical people I’ve spoken with who had the same reaction.

I think part of this is that in some domains it very rarely makes things up. If a kid uses for help with their history homework it will probably be 100% correct, because everything they ask it appears a thousand times in the training set.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#199

Post by rakah »

skaushik92 wrote: post #186
whimsicalism wrote: post #167

Earlier quoted context omitted.

> Based on my understanding of the approach behind ChatGPT, it is probably very close to a local maximum in terms of intelligence so we don't have to worry about the fearmongering spread by the "AI safety" people any time soon if AI research continues to follow this paradigm. I don't think you have a shred of evidence to back up this assertion.

… with confident language no less xD

Butlerian Jihad.
Post reply on HN