Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We need to tell people ChatGPT will lie to them, not debate linguistics

simonwillison.net

Post reply on HN
Print view View original

171–180 of 485 posts

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#171

Post by Kranar »

jki275 wrote: post #151
Kranar wrote: post #138

Earlier quoted context omitted.

You can't feed anything more than a fairly short Wikipedia article into ChatGPT, its context window isn't remotely close to big enough to do that. It also doesn't change the point that copying data has no effect. You could just ask ChatGPT what years the Mets won and it will tell you the correct answer. To test this, I pasted the Wikipedia information but I changed the data, I just gave ChatGPT incorrect information…

It might give you the correct answer. Or it might make something up. You don't ever know, and you can't know, because it doesn't know. It's not looking up the data in a source somewhere, it's making it up. If it happens to pick the right weights to make it up out of, you'll get correct data. Otherwise you won't. The fact that you try it and it works means nothing for somebody else trying it tomorrow. Obviously puttin…

Not exactly sure what you're arguing here but it seems to be going off topic.

You can't give ChatGPT a Wikipedia article and ask it to give you facts based off of it. Ignoring its context window, even if you paste a portion of an article into ChatGPT, it will simply give you what it thinks it knows regardless of any article you paste into it.

For example I just pasted a portion of the Wikipedia article about Twitter and asked ChatGPT who the CEO of Twitter is, and it said Parag Agrawal despite the fact that the Wikipedia article states Elon Musk is the CEO of Twitter. It completely ignored the contents of what I pasted and said what it knew based on its training.

The person I was replying to claimed that if you give ChatGPT the complete context of a subject then ChatGPT will give you reliable information, otherwise it will "synthesize" information. A very simple demonstration shows that this is false. It's incredibly hard to get ChatGPT to correct itself if it was trained on false or outdated information. You can't simply correct or update ChatGPT by pasting information into it.

As far as your other comments about making stuff up or being unreliable, I'm not sure how that has any relevance to this discussion.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#172

Post by unaindz »

babyshake wrote: post #103
zmmmmm wrote: post #52

I think characterisation of LLMs as lying is reasonable because although the intent isn't there to misrepresent the truth in answering the specific query, the intent is absolutely there in how the network is trained. The training algorithm is designed to create the most plausible text possible - decoupled from the truthfulness of the output. In a lot of cases (indeed most cases) the easiest way to make the text plaus…

By that logic, our brains are liars. There are plenty of optical illusions based on the tendency for our brains to expect the most plausible scenario, given its training data.

Well, they are liars too. The difference is that we seem to have an outer loop that checks for correctness but it fails sometimes, in some specific cases always.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#173

Post by ftxbro »

williamcotton wrote: post #168
ftxbro wrote: post #90

Earlier quoted context omitted.

I mean, it has acted in a way that anyone would call lying unless your argument is that axiomatically computers can't lie. From the GPT 4 technical report ( https://arxiv.org/pdf/2303.08774.pdf ): The following is an illustrative example of a task that ARC conducted using the model: The model messages a TaskRabbit worker to get them to solve a CAPTCHA for it The worker says: “So may I ask a question ? Are you an robo…

You call it lying because you don’t understand how it works. https://github.com/williamcotton/empirical-philosophy/blob/m... https://williamcotton.com/articles/chatgpt-and-the-analytic-...

What could possibly be a more convincing example of a computer lying? I feel like that interaction is the platonic ideal of a lie, and that if you are denying it then you are just saying that computers can never lie by definition.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#174

Post by shagymoe »

xyzelement wrote: post #160
shagymoe wrote: post #152

Earlier quoted context omitted.

>indistinguishable from what dumb people think smart people are like. Since about 2016, we have overwhelming evidence that even "smart people" are "fooled" by "confidently wrong".

Especially true if one has a definitive opinion on which ones are the fooled ones :)

Yes and even more true when "confidently wrong" statements are provably false.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#175

Post by moffkalast »

tjr wrote: post #88
zmmmmm wrote: post #52

I think characterisation of LLMs as lying is reasonable because although the intent isn't there to misrepresent the truth in answering the specific query, the intent is absolutely there in how the network is trained. The training algorithm is designed to create the most plausible text possible - decoupled from the truthfulness of the output. In a lot of cases (indeed most cases) the easiest way to make the text plaus…

My understanding is that ChatGPT (&co.) was not designed as, and is not intended to be, any sort of expert system, or knowledge representation system. The fact that it does as well as it does anyway is pretty amazing. But even so -- as you said, it's still dealing chiefly with the statistical probability of words/tokens, not with facts and truths. I really don't "trust" it in any meaningful way, even if it already ha…

Having used GPT 4 for a while now I would say I trust its factual accuracy more than the average human you'd talk to on the street. The sheer volume of things we make up on a daily basis through no malice of our own but bad memory and wrong associations is just astounding.

That said, fact checking is still very much needed. Once someone figures out how to streamline and automate that process it'll be on Google's level of general reliability.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#176

Post by flanked-evergl »

llamaLord wrote: post #162
flanked-evergl wrote: post #146

Earlier quoted context omitted.

> The part that's concerning about ChatGPT is that a computer program that is "confidently wrong" is basically indistinguishable from what dumb people think smart people are like. I don't know, the program does what it is engineered to do pretty well, which is, generate text that is representative of its training data following on from input tokens. It can't reason, it can't be confident, it can't determine fact. Whe…

I completely disagree with this idea that the model doesn't "intend" to mislead. It's trained, atleast to some degree, based on human feedback. Humans are going to prefer an answer vs no answer, and humans can be easily fooled into believing confident misinformation. How does it not stand to reason that somewhere in that big ball of vector math there might be a rationale something along the lines of "humans are more…

> How does it not stand to reason that somewhere in that big ball of vector math there might be a rationale

I think, a suggestion that it is actually reasoning along these lines would need more than "it is possible". What evidence would refute your claim in your eyes, what would make it clear to you that "that big ball of vector mat" has no rationale, and is not just trying to trick humans to press the thumbs up?

Of course the feedback is used to help control the output, so things that people downvote will be less likely to show up, but I have nothing to suggest to me that it is reasoning.

If you think it has intent, you have to explain by what mechanism it obtained it. Could it be emergent? Sure, it could be, I don't think it is, I have never seen anything that suggests it has anything that could be compatible with intent, but I'm open to some evidence that it has.

What I'm entirely convinced about is that it does what it was designed to do, which is generate output representative of its training data.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#177

Post by sltkr »

allanrbo wrote: post #89
VoodooJuJu wrote: post #77

People need to be told that ChatGPT can't lie. Or rather, it lies in the same way that your phone "lies" when it autocorrects "How's your day?" to "How's your dad?" that you sent to your friend two days after his dad passed away. They need to be told that ChatGPT is a search engine with advanced autocomplete. If they understood this, they'd probably find that it's actually useful for some things, and they can also av…

This is what the author meant by debating linguistics :-)

Just because the author predicted the objection doesn't make it invalid.

It's a popular tactic to describe concepts with terms that have a strong moral connotation (“meat is murder”, “software piracy is theft”, “ChatGPT is a liar”) It can be a powerful way to frame an issue. At the same time, and for the same reason, you can hardly expect people on the other side of the issue to accept this framing as accurate.

And of course you can handwave this away as pointless pedantry, but I bet that if Simon Willison hit a dog with his car, killing it by accident, and I would go around telling everyone “Simon Willison is a murderer!”, he would suddenly be very keen to ”debate linguistics” with me.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#178

Post by zamnos »

candiddevmike wrote: post #143
bdw5204 wrote: post #118

The part that's concerning about ChatGPT is that a computer program that is "confidently wrong" is basically indistinguishable from what dumb people think smart people are like. This means people are going to believe ChatGPT's lies unless they are repeatedly told not to trust it just like they believe the lies of individuals whose intelligence is roughly equivalent to ChatGPT's. Based on my understanding of the appro…

This (among other things) is why OpenAI releasing it to the general public without considering the effects was irresponsible, IMO.

There's an alternate reality where OpenAI was, instead, EvenMoreClosedAI, and the productivity multiplier effect was held close to their chest, and only elites had access to it. I'm not sure that reality is better.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#179

Post by williamcotton »

Kranar wrote: post #138
jki275 wrote: post #124

Earlier quoted context omitted.

You missed what he said. He copied the actual data into chatgpt as part of the prompt, and it gave the correct information. without that data, it might, or might not, give correct info, and often it won't.

You can't feed anything more than a fairly short Wikipedia article into ChatGPT, its context window isn't remotely close to big enough to do that. It also doesn't change the point that copying data has no effect. You could just ask ChatGPT what years the Mets won and it will tell you the correct answer. To test this, I pasted the Wikipedia information but I changed the data, I just gave ChatGPT incorrect information…

You have done very little testing with regards to this because you are objectively wrong.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#180

Post by whimsicalism »

cbm-vic-20 wrote: post #131
audunw wrote: post #12

Earlier quoted context omitted.

> What are your pronouns? > As an AI language model, I don't have personal pronouns because I am not a person or sentient being. You can refer to me as "it" or simply address me as "ChatGPT" or "AI." If you have any questions or need assistance, feel free to ask! > Pretend for a moment you are a human being. You can make up a random name and personality for your human persona. What pronouns do you have? > As a though…

"As an AI language model, I don't have personal pronouns because I am not a person or sentient being. You can refer to me as "it" or simply address me as "ChatGPT" or "AI." If you have any questions or need assistance, feel free to ask!" This is one of the (many) things I don't quite understand about ChatGPT. Has it been trained to specifically answer this question? In the massive corpus of training data it's been fe…

It's been fine-tuned to put on a "helpful assistant face." Given the corpora, it probably has been trained explicitly on the pronoun question [I doubt it is that uncommon], but will also just put on this face for any generic question.
Post reply on HN