Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We need to tell people ChatGPT will lie to them, not debate linguistics

simonwillison.net

Post reply on HN
Print view View original

151–160 of 485 posts

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#151

Post by jki275 »

Kranar wrote: post #138
jki275 wrote: post #124

Earlier quoted context omitted.

You missed what he said. He copied the actual data into chatgpt as part of the prompt, and it gave the correct information. without that data, it might, or might not, give correct info, and often it won't.

You can't feed anything more than a fairly short Wikipedia article into ChatGPT, its context window isn't remotely close to big enough to do that. It also doesn't change the point that copying data has no effect. You could just ask ChatGPT what years the Mets won and it will tell you the correct answer. To test this, I pasted the Wikipedia information but I changed the data, I just gave ChatGPT incorrect information…

It might give you the correct answer.

Or it might make something up.

You don't ever know, and you can't know, because it doesn't know. It's not looking up the data in a source somewhere, it's making it up. If it happens to pick the right weights to make it up out of, you'll get correct data. Otherwise you won't. The fact that you try it and it works means nothing for somebody else trying it tomorrow.

Obviously putting the data in and asking it the data is kind of silly. But you can put data into it and then ask it to provide more nuanced interpretations of the data you've given it, and it can do reasonably well doing that. People are using it to debug code, I've personally used the ghidra plugins to good effect -- the way that works is to feed in the whole function and then have chatgpt tell you what it can deduce about it. It generally provides reasonably useful interpretations.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#152

Post by shagymoe »

bdw5204 wrote: post #118

The part that's concerning about ChatGPT is that a computer program that is "confidently wrong" is basically indistinguishable from what dumb people think smart people are like. This means people are going to believe ChatGPT's lies unless they are repeatedly told not to trust it just like they believe the lies of individuals whose intelligence is roughly equivalent to ChatGPT's. Based on my understanding of the appro…

>indistinguishable from what dumb people think smart people are like.

Since about 2016, we have overwhelming evidence that even "smart people" are "fooled" by "confidently wrong".

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#153

Post by sltkr »

cbm-vic-20 wrote: post #131
audunw wrote: post #12

Earlier quoted context omitted.

> What are your pronouns? > As an AI language model, I don't have personal pronouns because I am not a person or sentient being. You can refer to me as "it" or simply address me as "ChatGPT" or "AI." If you have any questions or need assistance, feel free to ask! > Pretend for a moment you are a human being. You can make up a random name and personality for your human persona. What pronouns do you have? > As a though…

"As an AI language model, I don't have personal pronouns because I am not a person or sentient being. You can refer to me as "it" or simply address me as "ChatGPT" or "AI." If you have any questions or need assistance, feel free to ask!" This is one of the (many) things I don't quite understand about ChatGPT. Has it been trained to specifically answer this question? In the massive corpus of training data it's been fe…

Why is that surprising? GPT-4 is clearly smart enough to know that inanimate objects are referred to as "it", since it's keenly aware it is an AI language model, it would also apply that pronoun to itself.

You have to realize that GPT is fundamentally just a token predictor. It has been primed with some script (provided by OpenAI) to which the user input is added. For example:

    The following is a dialogue between a user and a computer assistant called ChatGPT. ChatGPT is an AI language model that tries to be helpful and informative, while avoiding misinformation and offensive language. ChatGPT typically replies with two to five sentences.

    User: Do you like cake?
    ChatGPT: As an AI language model, I do not need to eat.
    User: What are your pronouns?
    ChatGPT:
It then generates a sequence of tokens based on the context and its general knowledge. It seems only logical that it would generate a reply like it does. That's what you or I would do, isn't it? And neither of us have been trained to do so.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#154

Post by danpat »

flanked-evergl wrote: post #146
bdw5204 wrote: post #118

The part that's concerning about ChatGPT is that a computer program that is "confidently wrong" is basically indistinguishable from what dumb people think smart people are like. This means people are going to believe ChatGPT's lies unless they are repeatedly told not to trust it just like they believe the lies of individuals whose intelligence is roughly equivalent to ChatGPT's. Based on my understanding of the appro…

> The part that's concerning about ChatGPT is that a computer program that is "confidently wrong" is basically indistinguishable from what dumb people think smart people are like. I don't know, the program does what it is engineered to do pretty well, which is, generate text that is representative of its training data following on from input tokens. It can't reason, it can't be confident, it can't determine fact. Whe…

> but why would you expect that from it?

If you understand what it is doing, then you don't. But the layman will just see a computer that talks in language they understand, and will infer intent and sentience are behind that, because that's the only analog they have for a thing that can talk back to them with words that appear to make sense at the complexity level that ChatGPT is achieving.

Most humans do not have sufficient background to understand what they're really being presented with, they will take it at face value.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#155

Post by kozikow »

shepardrtc wrote: post #10

Someone in my company spent the past month setting up ChatGPT to work with our company's knowledge base. Not by a plugin or anything, just by telling ChatGPT where to find it. They didn't believe that ChatGPT was making any of it up, just that sometimes it got it wrong. I stopped arguing after a while.

I was able to create a GPT-4 based bot initally based on knowledge base that provides accurate information. To do this, I first converted a knowledge base article into a question and answer (Q&A) format using GPT-4 - quick explanation and article link if necessary. Then, I used the API to generate more Q&A pairs by asking GPT-4 to predict what users might ask and create corresponding answers.

On my side I now search for the most relevant Q&A pair based on the embedding of user's input and QA and jam as much as I can into the token limit. It provides accurate answers 99% of the time. If it can't find a suitable answer, it may create a plausible response on the spot, but that's getting rarer as training set grows.

To prevent the bot from providing incomplete information, you can instruct it to ask users to contact support via email if it doesn't have enough information - either prompt engineering or examples in training set. Alternatively, you can have the bot insert a token like "%%TICKET%%" which you can later use to open a support ticket, summarizing the conversation and attaching relevant chat history, logs, etc.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#156

Post by Herval_freire »

bdw5204 wrote: post #118

The part that's concerning about ChatGPT is that a computer program that is "confidently wrong" is basically indistinguishable from what dumb people think smart people are like. This means people are going to believe ChatGPT's lies unless they are repeatedly told not to trust it just like they believe the lies of individuals whose intelligence is roughly equivalent to ChatGPT's. Based on my understanding of the appro…

[dead]

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#157

Post by flanked-evergl »

zug_zug wrote: post #27
retrac wrote: post #2

Indeed. We are anthropomorphizing them. I do it all the time and I should know better. There are already a few reports floating around of people who have seemingly been driven mad, come to believe strongly that the language model they're using is a conversation with a real person. A lot of people will really struggle with this going forward, I think. If we're going to anthropomorphize, then let us anthropomorphize wi…

That all feels tad dramatic. It's like a person on the internet -- in that it's wrong 20% of the time, often confidently so. But the distinction is it's less rude, and more knowledgeable.

> It's like a person on the internet -- in that it's wrong 20% of the time, often confidently so. But the distinction is it's less rude, and more knowledgeable.

Do you think if it is trained on only factual content, it will only say factual things? How does that even really work? Is there research on this? How does it then work for claims that are not factual, like prescriptive statements? And what about fiction? Will it stop being able to write prose? What if I create new facts?

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#158

Post by ojosilva »

> Should we warn people off or help them on?

"We" and "people" here are idealizations, just like we idealize LLMs as thinking entities.

"We" can't warn "people".

ChatGPT is a tool, and users will be users. Yes, one can label output as potentially harmful, false and deceiving. But, just like tobacco and its warnings, people still will peruse the tool because the tool does what it does, it responds in context to prompts and, even knowing it's imperfect, humans are imperfect and tend to swallow it whole. We need to push forward ours topics at hand, we desire that input, that mirroring. So now, either the tool improves its quality and accuracy or something new will have to come along. Then we can move on and forget we ever did tell ChatGPT about our problems.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#159

Post by red75prime »

hgsgm wrote: post #111
red75prime wrote: post #85

Earlier quoted context omitted.

> Ask the question: given improbable but thruthful output but plausible untruthful output, which does the network choose? "Plausible" means "that which the majority of people is likely to say". So, yes, a foundational model is likely to say the plausible thing. On the other hand, it has to have a way to output a truthful answer too, to not fail on texts produced by experts. So, it's not impossible that the model coul…

> "that which the majority of people is likely to say" .. and saying "I don't know" is forbidden by the programmers. That is a huge part of the problem.

I guess it's not that straightforward. It's probably a combination of much less prevalent use of "don't know" online, low scores of "don't know" in RLHF, system prompt instructing GPT to give helpful responses, and, yeah, maybe token sampling algorithm is tuned to disfavor explicitly uncertain responses.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#160

Post by xyzelement »

shagymoe wrote: post #152
bdw5204 wrote: post #118

The part that's concerning about ChatGPT is that a computer program that is "confidently wrong" is basically indistinguishable from what dumb people think smart people are like. This means people are going to believe ChatGPT's lies unless they are repeatedly told not to trust it just like they believe the lies of individuals whose intelligence is roughly equivalent to ChatGPT's. Based on my understanding of the appro…

>indistinguishable from what dumb people think smart people are like. Since about 2016, we have overwhelming evidence that even "smart people" are "fooled" by "confidently wrong".

Especially true if one has a definitive opinion on which ones are the fooled ones :)
Post reply on HN