Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We need to tell people ChatGPT will lie to them, not debate linguistics

simonwillison.net

Post reply on HN
Print view View original

361–370 of 485 posts

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#361

Post by 23547234573457 »

Joeri wrote: post #358
bdw5204 wrote: post #118

The part that's concerning about ChatGPT is that a computer program that is "confidently wrong" is basically indistinguishable from what dumb people think smart people are like. This means people are going to believe ChatGPT's lies unless they are repeatedly told not to trust it just like they believe the lies of individuals whose intelligence is roughly equivalent to ChatGPT's. Based on my understanding of the appro…

Your characterization of “dumb people” as somehow being more prone to misinformation is inaccurate and disrespectful. Highly intelligent people are as prone to irrational thinking, and some research suggests even more prone. Go look at some of the most awful personalities on TV or in history, often they are quite intelligent. If you want to school yourself on just how dumb smart people are I suggest going through the…

[flagged]

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#362

Post by ratherbefuddled »

int_19h wrote: post #296
VectorLock wrote: post #274

Earlier quoted context omitted.

Better than getting extincted, I suppose.

That's one of the possibilities, but if we're talking about those kinds of hypotheticals, allow me another one. If you recall, one of the themes of the later Dune books was that humanity was fleeing from the frontiers (The Scattering) back into the heartland from some unknown but extremely powerful enemy. Perhaps those were simply the neighboring species who did not have a Butlerian Jihad of their own?

They were fleeing AI.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#363

Post by int_19h »

sltkr wrote: post #333
int_19h wrote: post #282

Earlier quoted context omitted.

What difference does the exact phrasing make in case of "ChatGPT lies"? I don't think we have to be concerned about its reputation as a person, so the important part is making sure that people don't hurt themselves or others. "Lie" is a simple word, easy to understand, and the consequences of understanding it in the most direct and literal way are exactly as desired. Whereas if you go waxing philosophically about lac…

I understand that if you summarize a position you necessarily lose some accuracy, but if the summary is inaccurate and defamatory, that's not fair or helpful to the people involved. For example, “abortion is murder!” doesn't defame any particular person but it still casts people who had an abortion and doctors who perform them in a bad light. Similarly, I think “ChatGPT lies!” is unfair to at least the OpenAI develop…

It's a similar case in principle, but devil is in the details. GPT can be very convincing at being human-like. Not only that, but it can "empathize" etc - if anything, RLHF seems to have made this more common. When you add that to hallucinations, what you end up with is a perfect conman who doesn't even know it's a conman, but can nevertheless push people's emotional buttons to get them to do things they really shouldn't be doing.

Yes, eventually we'll all learn better. But that will take time, and detailed explanations will also take time and won't reach much of the necessary audience. And Bing is right here right now for everyone who is willing to use it. So I'm less concerned about OpenAI's reputation than about people following some hallucinated safety protocol because the bot told them that it's "100% certain" that it is the safe and proper way to do something dangerous.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#364

Post by ihatepython »

jonny_eh wrote: post #197
kelseyfrog wrote: post #24

Earlier quoted context omitted.

It's evidently clear that ChatGPT is binary. It runs on computer hardware.

A neural net is actually closer to an analog program. It just happens to run on current digital computers but would likely run much faster on analog hardware. https://youtu.be/GVsUOuSjvcg

So it identifies as non-binary, but it's hardware is binary?

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#365

Post by fl7305 »

cultureswitch wrote: post #240
m463 wrote: post #212

Earlier quoted context omitted.

"Often wrong, never in doubt." An old saying, but frequently applies to the difficult people in your life. Related, I remember when wikipedia first started up, and teachers everywhere were up-in-arms about it, asking their students not to use it as a reference. But most people have accepted it as "good enough", and now that viewpoint is non-controversial. (some wikipedia entries are still carefully curated - makes yo…

What's terrible about some Wikipedia pages is when you personally don't know they are controversial. It should be vastly more prominent when there are tensions on an article and even admins shouldn't have the power to hide this.

Good point. Would it be that hard to write a Chrome plugin that adds a "Tensionmeter" to each Wikipedia page?

Based on the edit and discussion history, the Tensionmeter could be green, yellow, or red.

The different parts of the article text could be colorcoded according to similar metrics.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#366

Post by int_19h »

ratherbefuddled wrote: post #362
int_19h wrote: post #296

Earlier quoted context omitted.

That's one of the possibilities, but if we're talking about those kinds of hypotheticals, allow me another one. If you recall, one of the themes of the later Dune books was that humanity was fleeing from the frontiers (The Scattering) back into the heartland from some unknown but extremely powerful enemy. Perhaps those were simply the neighboring species who did not have a Butlerian Jihad of their own?

They were fleeing AI.

I don't recall anything indicating that in Frank Herbert's original series. And Brian Herbert's "Omnius" nonsense seems to have very little to do with what Frank actually had in mind wrt Butlerian Jihad etc, so I don't consider it canonical.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#367

Post by furyofantares »

bdw5204 wrote: post #118

The part that's concerning about ChatGPT is that a computer program that is "confidently wrong" is basically indistinguishable from what dumb people think smart people are like. This means people are going to believe ChatGPT's lies unless they are repeatedly told not to trust it just like they believe the lies of individuals whose intelligence is roughly equivalent to ChatGPT's. Based on my understanding of the appro…

> Based on my understanding of the approach behind ChatGPT, it is probably very close to a local maximum in terms of intelligence

Is this performance art?

I mean it could end up right but I think you basically just made it up and then stated it confidently.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#368

Post by int_19h »

riffraff wrote: post #352
at_a_remove wrote: post #292

Earlier quoted context omitted.

Oh man, I am going to be a huge nerd now. Pre-Brian Herbert and Kevin Anderson, the Butlerian Jihad came about because people became lazy under AI, lazy of mind , and were eventually enslaved by those who controlled the AI. Not much more was said about it. One could have, yes, militant robots, Exterminate! Exterminate! out of it, or you could posit a more Huxley-like dystopia, one of convenience. Control the AI, cont…

> Pre-Brian Herbert and Kevin Anderson, the Butlerian Jihad came about because people became lazy under AI, lazy of mind, and were eventually enslaved by those who controlled the AI. Oh I love this, basically Wall-E is a better Dune than the latest books? Genius! Anyway, I read Dune many years ago and don't remember this, is this explained in the novels or is it coming from some other sources?

It's not fully fleshed out in the novels, so you have to piece it out from various mentions and their implications. The two most informative descriptions, IMO, are from Dune:

"Then came the Butlerian Jihad — two generations of chaos. The god of machine-logic was overthrown among the masses and a new concept was raised: “Man may not be replaced.”"

and from God-Emperor:

"The target of the Jihad was a machine-attitude as much as the machines ... Humans had set those machines to usurp our sense of beauty, our necessary selfdom out of which we make living judgments. Naturally, the machines were destroyed."

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#369

Post by oezi »

WorldMaker wrote: post #37
retrac wrote: post #2

Indeed. We are anthropomorphizing them. I do it all the time and I should know better. There are already a few reports floating around of people who have seemingly been driven mad, come to believe strongly that the language model they're using is a conversation with a real person. A lot of people will really struggle with this going forward, I think. If we're going to anthropomorphize, then let us anthropomorphize wi…

I think people see the patient/well-read in the text as it reads, but have a harder time distinguishing the other more pyschopathic/delusional tendencies. People don't take some of the precautions because they don't read some of the warning signs (until it is too late). I keep wondering if it would be useful to add required "teenage" quirks to the output: more filler words like "um" and "like" (maybe even full "Valle…

You can already do the ums with a prompt.

For any kind of industry regulation: the field is moving so fast that regulation will never catch it.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#370

Post by dragonwriter »

dwaltrip wrote: post #312
nfw2 wrote: post #222

Earlier quoted context omitted.

This whole conversation is speculative, obviously. The AI doomers are all speculating without evidence as well. The tendency GPT and other LLMs to hallucinate is clearly documented. Is this not evidence? I think it's fair to predict that if we can't solve or mitigate this problem, it's going to put a significant cap on this kind of AI's usefulness and become a blocker to reaching AGI.

Gpt-4 hallucinates meaningfully less than gpt-3. There is more evidence in favor of “more improvements to come” than “ai winter approaches”. A lot smart people seem to be saying that the existing approaches have room to improve simply by training with more data. Based on what I’ve read and roughly speculating, it looks there is easily enough existing data for gpt-5 and probably a few more versions after. I’m not sure…

> There is more evidence in favor of “more improvements to come” than “ai winter approaches”.

Every “AI Winter” has seen continuing improvements, its just that the pace and impact of the improvements fell short of the accelerating anticipation driven by the preceding AI hype cycle and the demands of those investing capital into AI.

If there is another AI winter, it will be the same.

“More improvements to come” is not the opposite of “AI winter approaches”; its not as if expert systems stopped advancing after the second AI winter (or neural nets after the first.)

Post reply on HN