Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We need to tell people ChatGPT will lie to them, not debate linguistics

simonwillison.net

Post reply on HN
Print view View original

201–210 of 485 posts

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#201

Post by musha68k »

zuminator wrote: post #187
llamaLord wrote: post #162

Earlier quoted context omitted.

I completely disagree with this idea that the model doesn't "intend" to mislead. It's trained, atleast to some degree, based on human feedback. Humans are going to prefer an answer vs no answer, and humans can be easily fooled into believing confident misinformation. How does it not stand to reason that somewhere in that big ball of vector math there might be a rationale something along the lines of "humans are more…

I don't think it intends to mislead because its answers are probabilistic. It's designed to distill a best guess out of data which is almost certain to be incomplete or conflicting. As human beings we do the same thing all the time. However we have real life experience of having our best guesses bump up against reality and lose. ChatGPT can't see reality. It only knows what "really being wrong" is to the extent that…

At the current time it is flawed to the point of being dangerous. It’s a - sometimes - useful new set of tooling. I guess there is value in that … but does it outweigh the always lurking “bullshit generator” bad parts? I’m not sure.

It’s interesting to watch the developments though - like a fire - one just doesn’t fully know what the flames are consuming.. yet.

Maybe through all the new training data we collectively provide for free, it will get better? Maybe not though, maybe it will just get better at bullshitting?

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#202

Post by sebzim4500 »

kazinator wrote: post #183

> This is a serious bug that has so far resisted all attempts at a fix. Does ChatGPT have a large test suite consisting of a large number of input question and expected responses that have to match? Or an equivalent specification? If not, there can be no "bug" in ChatGPT.

>Does ChatGPT have a large test suite consisting of a large number of input question and expected responses that have to match?

They are trying to crowdsource this with OpenAI evals. https://github.com/openai/evals

I'm sure they have a lot of internal benchmarks too, but of course they won't share them.

>If not, there can be no "bug" in ChatGPT.

I don't understand the objection. Are you claiming that bugs only exist if you have testcases for them?

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#203

Post by hackinthebochs »

checkyoursudo wrote: post #115
adroniser wrote: post #94

Earlier quoted context omitted.

[flagged]

How so? GPT LLM algorithms use a probabilistic language model to generate text. It is trained on a large corpus of text data, and it estimates the probability distribution of the next word given the previous words in the sequence. The algorithm tokenizes the input into a sequence of tokens and then generates the next token(s) in the sequence based on the probabilities learned during training. These probabilities are…

>The algorithm tokenizes the input into a sequence of tokens and then generates the next token(s) in the sequence based on the probabilities learned during training.

This is just a description of the input/output boundary of the system. The question is what goes on in-between and how to best characterize it? My input/output is also "tokens" of a sort: word units, sound patterns, color patches, etc.

>Your ideas cohere from network connections between the neurons in your brain, and then you come up with words to match your idea, not your previous words or the frequency that the words appear in your brain.

At some high level of description, yes. At a low level it's just integrating action potentials. What's to say there isn't a similar high level of description that characterizes the process by which ChatGPT decides on its continuation?

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#204

Post by [deleted] »

whimsicalism wrote: post #167
bdw5204 wrote: post #118

The part that's concerning about ChatGPT is that a computer program that is "confidently wrong" is basically indistinguishable from what dumb people think smart people are like. This means people are going to believe ChatGPT's lies unless they are repeatedly told not to trust it just like they believe the lies of individuals whose intelligence is roughly equivalent to ChatGPT's. Based on my understanding of the appro…

> Based on my understanding of the approach behind ChatGPT, it is probably very close to a local maximum in terms of intelligence so we don't have to worry about the fearmongering spread by the "AI safety" people any time soon if AI research continues to follow this paradigm. I don't think you have a shred of evidence to back up this assertion.

[deleted]

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#205

Post by tlear »

Interesting examples of what he thinks it is helpful with. For example the API one, how do you know that it did not make up the decorator example out of thin air and it is not garbage? You have to google it or know it beforehand(he probably knew it already).

For me the cognitive load of second guessing everything is just too painful.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#206

Post by mLuby »

Telling kids not to play with loaded guns is like telling adults and companies not to trust ChatGPT's answers.

Yes, it's better than doing absolutely nothing. And yes, in the strictest sense it's their fault if they ignore the warning and accidentally use it to cause harm.

But the blame really belongs with the adults in the room for allowing such an irresponsible situation in the first place.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#207

Post by eterevsky »

mLuby wrote: post #206

Telling kids not to play with loaded guns is like telling adults and companies not to trust ChatGPT's answers. Yes, it's better than doing absolutely nothing. And yes, in the strictest sense it's their fault if they ignore the warning and accidentally use it to cause harm. But the blame really belongs with the adults in the room for allowing such an irresponsible situation in the first place.

So in this analogy, what adult should be doing? Banning LLMs?

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#208

Post by kenjackson »

njarboe wrote: post #169
zmmmmm wrote: post #52

I think characterisation of LLMs as lying is reasonable because although the intent isn't there to misrepresent the truth in answering the specific query, the intent is absolutely there in how the network is trained. The training algorithm is designed to create the most plausible text possible - decoupled from the truthfulness of the output. In a lot of cases (indeed most cases) the easiest way to make the text plaus…

I think it is much closer to bullshit. The bullshitter cares not to tell truth or deceive, just to sound like they know what they are talking about. To impress. Seems like ChatGPT to a T.

ChatGPT no more cares to impress or persuade you as it cares to tell the truth or lie. It will say what its training maps to its model the best. No more, no less. If you believe it or not -- ChatGPT doesn't care -- except to the extent that you report the answer and they tweak its training/model in the future.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#209

Post by toss1 »

SanderNL wrote: post #98
toss1 wrote: post #75

Earlier quoted context omitted.

ChatGPT4 is the human equivalent of a primate or apex predator encountering a mirror for the first time in their lives. ChatGPT4 is reflecting back at us an extract of the sum of the human output it has been 'trained' upon. Of course the output feels human! LLMs have zero capability to abstract anything resembling a concept, to abstract a truth from a fiction, or to reason about such things. The generation of the mos…

Lying implies a hefty dose of anthropomorphism. Lying means a lot of things, all of which have to do with intelligence. By telling it lies you actually make it seem more intelligent. > human intelligence is not merely pattern matching Citation needed

I agree that lying also has a connotation of intent and therefore intelligence, and so is at least technically inappropriate.

Yet I agree with the researchers that the benefits of using "lying" instead of the more technically accurate "confabulation" outweigh the negatives. In communicating with new users who may over-rely on it, getting across highly technically correct metaphors is less important than delivering the message that "you will receive incorrect information from this thing". The most to-the-point way is "It's going to lie to you".

>Citation needed Studied logic, language, neuroscience, and computing in college. There are entire sybsystems of the brain and perceptual system primarily filtering information, as well as the ability to abstract concepts away from the patterns and draw inferences, it goes on an on. I could go on, but "It's all pattern matching" is an over-reductive argument that accounts for very little from either the perceptual system or behavioral system (or it's expanded and goalposts moved so it's unfalsifiable, therefore not even wrong).

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#210

Post by arvidkahl »

I consider ChatGPT to be a gaslighting engine at scale. Every word it "utters" is meant to sound believable and convincing. It doesn't know truth or fact, just likelihood of a string of text tokens being believable.

I've started explaining it in terms of a "conman" to my friends & family. It will say anything to make you think it's right. It will even apologize for making a mistake if you insist that 2+2 is 5. That's what a liar would do to make you look good. (That's usually when people get it.)

Post reply on HN