Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We need to tell people ChatGPT will lie to them, not debate linguistics

simonwillison.net

Post reply on HN
Print view View original

281–290 of 485 posts

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#281

Post by whimsicalism »

int_19h wrote: post #269
whimsicalism wrote: post #180

Earlier quoted context omitted.

It's been fine-tuned to put on a "helpful assistant face." Given the corpora, it probably has been trained explicitly on the pronoun question [I doubt it is that uncommon], but will also just put on this face for any generic question.

This stuff is not fine-tuning - it's RLHF (reinforcement learning from human feedback). Basically just a lot of people marking up responses as good / bad according to the assessment criteria in the script they're given. And yes, it is very likely that it would have seen that exactly question in RLHF. But even if not, it had seen enough to broadly "understand" what kinds of topics are sensitive and how to tiptoe aroun…

RLHF is fine tuning :)

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#282

Post by int_19h »

sltkr wrote: post #177
allanrbo wrote: post #89

Earlier quoted context omitted.

This is what the author meant by debating linguistics :-)

Just because the author predicted the objection doesn't make it invalid. It's a popular tactic to describe concepts with terms that have a strong moral connotation (“meat is murder”, “software piracy is theft”, “ChatGPT is a liar”) It can be a powerful way to frame an issue. At the same time, and for the same reason, you can hardly expect people on the other side of the issue to accept this framing as accurate. And o…

What difference does the exact phrasing make in case of "ChatGPT lies"? I don't think we have to be concerned about its reputation as a person, so the important part is making sure that people don't hurt themselves or others. "Lie" is a simple word, easy to understand, and the consequences of understanding it in the most direct and literal way are exactly as desired. Whereas if you go waxing philosophically about lack of agency etc, you lose most of the audience before you get to the point. This is not about intellectual rigor and satisfaction - it's about a very real and immediate public safety concern.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#283

Post by vkou »

m463 wrote: post #212
bdw5204 wrote: post #118

The part that's concerning about ChatGPT is that a computer program that is "confidently wrong" is basically indistinguishable from what dumb people think smart people are like. This means people are going to believe ChatGPT's lies unless they are repeatedly told not to trust it just like they believe the lies of individuals whose intelligence is roughly equivalent to ChatGPT's. Based on my understanding of the appro…

"Often wrong, never in doubt." An old saying, but frequently applies to the difficult people in your life. Related, I remember when wikipedia first started up, and teachers everywhere were up-in-arms about it, asking their students not to use it as a reference. But most people have accepted it as "good enough", and now that viewpoint is non-controversial. (some wikipedia entries are still carefully curated - makes yo…

There's a million reasons for why Wikipedia is usually a good source of information, without being a good reference.

You know what can be a good reference? One or more of the references that Wikipedia cites at the bottom of the page.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#284

Post by worrycue »

DebtDeflation wrote: post #139
zmmmmm wrote: post #52

I think characterisation of LLMs as lying is reasonable because although the intent isn't there to misrepresent the truth in answering the specific query, the intent is absolutely there in how the network is trained. The training algorithm is designed to create the most plausible text possible - decoupled from the truthfulness of the output. In a lot of cases (indeed most cases) the easiest way to make the text plaus…

IMO, it just requires the same level of skepticism as a Google search. Just because you enter a query into the search bar and Google returns a list of links and you click one of those links and it contains content that makes a claim, doesn't mean that claim is correct. After all, this is largely what GPT has been trained on.

Seems worse to be frank.

The webpages Google search delivers might be filled with falsehood but google search itself does its job of finding said pages which contain the terms you inputted fairly reliably.

With GPT, not only there’s a chance its training data is full of falsehood, you can add the possibility of it inventing “original” falsehoods on top of that.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#285

Post by knome »

flanked-evergl wrote: post #196
zuminator wrote: post #187

Earlier quoted context omitted.

I don't think it intends to mislead because its answers are probabilistic. It's designed to distill a best guess out of data which is almost certain to be incomplete or conflicting. As human beings we do the same thing all the time. However we have real life experience of having our best guesses bump up against reality and lose. ChatGPT can't see reality. It only knows what "really being wrong" is to the extent that…

> Even with our advantage of interacting with the real world, I'd still wager that the average person's no better (and probably worse) than ChatGPT for uttering factual truth. ChatGPT makes up non-existing APIs for Google cloud and Go out of whole cloth. I have never met a human who does that. If we reduce it down to how often most people are wrong vs how often ChatGPT is wrong, then sure, people may be on average wr…

>I have never met a human who does that.

Schizophrenics tend to be unable to tell the difference between their delusions and reality.

On a less extreme note, I've known plenty of humans that constantly make up details and rewrite stories of events as well. They are usually very confident that their retelling is accurate, even when presented with evidence that they have reimagined portions of it.

>It can't reason, it can't be confident, it can't determine fact.

In the following link I tasked it with having to generate novel metaphors that have an equivalent non-literal meaning as an first set, changing the literal topics while maintaining the non-literal topics.

https://news.ycombinator.com/item?id=35392025

How would you suggest it does this without reason? To hand-wave what it can do as "merely generating the next token statistically" seems like a gross understatement. I doubt it picked up a corpus of car-to-curling metaphor translations somewhere :P

I understand how chatgpt is creating its next tokens, but I have my doubts that the process should be viewed as unreasoning. GPT-3 had 96 layers and billions of weights between them. GPT-4 increases on this even further. GPT-5, which I've seen mentioned as currently training, will no doubt once again expand this range.

It is not a human reasoning, certainly. It has no experiential data to draw on, yes. No experiences to root its metaphoric language as we humans use. But without reason, how does it translate between abtractions?

It's terrible at math, yes. But it lacks any capacity for "visualization" or "using a board in its head" or "working through a problem by moving things around in its head". It doesn't have any equivalent to the portions of our brains that handle such things.

But humans too can suffer dyscalulia if a specific portion of the brain is injured.

I expect that we are dealing with what amounts to a fairly brain-damaged intelligence. It seems capable of abstract metaphoric reasoning, with many other sorts of reasoning being denied to it by the nature of how we created it.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#286

Post by IIAOPSW »

flanked-evergl wrote: post #146
bdw5204 wrote: post #118

The part that's concerning about ChatGPT is that a computer program that is "confidently wrong" is basically indistinguishable from what dumb people think smart people are like. This means people are going to believe ChatGPT's lies unless they are repeatedly told not to trust it just like they believe the lies of individuals whose intelligence is roughly equivalent to ChatGPT's. Based on my understanding of the appro…

> The part that's concerning about ChatGPT is that a computer program that is "confidently wrong" is basically indistinguishable from what dumb people think smart people are like. I don't know, the program does what it is engineered to do pretty well, which is, generate text that is representative of its training data following on from input tokens. It can't reason, it can't be confident, it can't determine fact. Whe…

In short, it is not a liar its a bullshitter. A liar misrepresents facts, a bullshitter doesn't care if what they say is true so long as they pass in conversation.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#287

Post by worrycue »

zmmmmm wrote: post #52

I think characterisation of LLMs as lying is reasonable because although the intent isn't there to misrepresent the truth in answering the specific query, the intent is absolutely there in how the network is trained. The training algorithm is designed to create the most plausible text possible - decoupled from the truthfulness of the output. In a lot of cases (indeed most cases) the easiest way to make the text plaus…

Whatever the intent or lack of, the result is the same, people are being given incorrect information.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#288

Post by int_19h »

MikePlacid wrote: post #234

ChatGPT (often called Geptile in Russian - from “heptile”, which is a very powerful but very dangerous rocket fuel) can well lie when debating linguistics, lol. Like: Например, в слове "bed" ударение на первом слоге, а в слове "get" - на втором. Here, Geptile (in a good Russian) insists that English word “get” has two syllables! When pointed to the error, Geptile apologizes - and then repeats the error again. But I g…

If you want real fun, ask it about the phonetics of something reasonably obscure. You might find out wonderful new things, like /s/ being an affricate in a language that doesn't even have affricates.

It's also quite confident that it can speak seemingly just about any language that occurs in its dataset. If you ask it to do that with e.g. Old Norse or Lojban, hilarity ensues, especially if you keep pointing out things that are wrong or just don't make any sense.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#289

Post by bsder »

worrycue wrote: post #277
cultureswitch wrote: post #236

Earlier quoted context omitted.

GPT-2, 3 and 4 keep on showing that increasing the size of the model keeps on making the results better without slowing down. This is remarkable, because usually in practical machine learning applications there is a quickly reached plateau of effectiveness beyond which a bigger model doesn't yield better results. With these ridiculously huge LLMs, we're not even close yet. And this was exciting news in papers from ye…

Yet they still can’t get it to shut the hell up if it doesn’t know and to not make shit up to pad its answers.

"I don't know" isn't in the training set, after all. Welcome to Eternal September on the Internet.

The first "AI" that actually says "I don't know" in response to a question will get my attention.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#290

Post by yosito »

bdw5204 wrote: post #118

The part that's concerning about ChatGPT is that a computer program that is "confidently wrong" is basically indistinguishable from what dumb people think smart people are like. This means people are going to believe ChatGPT's lies unless they are repeatedly told not to trust it just like they believe the lies of individuals whose intelligence is roughly equivalent to ChatGPT's. Based on my understanding of the appro…

> stupid people might get their brains programmed by AI rather than by demagogues which should have little practical difference

Until the demagogues train the AI.

Post reply on HN