Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We need to tell people ChatGPT will lie to them, not debate linguistics

simonwillison.net

Post reply on HN
Print view View original

131–140 of 485 posts

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#131

Post by cbm-vic-20 »

audunw wrote: post #12
ihatepython wrote: post #8

Earlier quoted context omitted.

Anybody know what are ChatGPT's pronouns?

> What are your pronouns? > As an AI language model, I don't have personal pronouns because I am not a person or sentient being. You can refer to me as "it" or simply address me as "ChatGPT" or "AI." If you have any questions or need assistance, feel free to ask! > Pretend for a moment you are a human being. You can make up a random name and personality for your human persona. What pronouns do you have? > As a though…

"As an AI language model, I don't have personal pronouns because I am not a person or sentient being. You can refer to me as "it" or simply address me as "ChatGPT" or "AI." If you have any questions or need assistance, feel free to ask!"

This is one of the (many) things I don't quite understand about ChatGPT. Has it been trained to specifically answer this question? In the massive corpus of training data it's been feed, has it encountered a similar series of tokens that would cause it to produce this output?

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#132

Post by gwright »

ChatGTP wrote: post #101
tjr wrote: post #88

Earlier quoted context omitted.

My understanding is that ChatGPT (&co.) was not designed as, and is not intended to be, any sort of expert system, or knowledge representation system. The fact that it does as well as it does anyway is pretty amazing. But even so -- as you said, it's still dealing chiefly with the statistical probability of words/tokens, not with facts and truths. I really don't "trust" it in any meaningful way, even if it already ha…

So do you find it alarming that people are trying to give such a system “agency” ?

I do. I think the anthropomorphic language that people use to describe these systems is inaccurate and misleading. An Australian mayor has claimed that ChatGPT "defamed" him. The title of this article says that we should teach people that text generation tools "lie". Other articles suggest that ChatGPT "knows" things.

It is extremely interesting to me how much milage can be gotten out of an LLM by observing patterns in text and generating similar patterns.

It is extremely frustrating to me to see how easily people think that this is evidence of intelligence or knowledge or "agency" as you suggest.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#133

Post by [deleted] »

ChatGTP wrote: post #101
tjr wrote: post #88

Earlier quoted context omitted.

My understanding is that ChatGPT (&co.) was not designed as, and is not intended to be, any sort of expert system, or knowledge representation system. The fact that it does as well as it does anyway is pretty amazing. But even so -- as you said, it's still dealing chiefly with the statistical probability of words/tokens, not with facts and truths. I really don't "trust" it in any meaningful way, even if it already ha…

So do you find it alarming that people are trying to give such a system “agency” ?

[deleted]

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#134

Post by te_chris »

ChatGTP wrote: post #101
tjr wrote: post #88

Earlier quoted context omitted.

My understanding is that ChatGPT (&co.) was not designed as, and is not intended to be, any sort of expert system, or knowledge representation system. The fact that it does as well as it does anyway is pretty amazing. But even so -- as you said, it's still dealing chiefly with the statistical probability of words/tokens, not with facts and truths. I really don't "trust" it in any meaningful way, even if it already ha…

So do you find it alarming that people are trying to give such a system “agency” ?

Yes. It’s very weird seeing the religiosity surrounding this whole thing.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#136

Post by vernon99 »

bdw5204 wrote: post #118

The part that's concerning about ChatGPT is that a computer program that is "confidently wrong" is basically indistinguishable from what dumb people think smart people are like. This means people are going to believe ChatGPT's lies unless they are repeatedly told not to trust it just like they believe the lies of individuals whose intelligence is roughly equivalent to ChatGPT's. Based on my understanding of the appro…

> Based on my understanding of the approach behind ChatGPT, it is probably very close to a local maximum in terms of intelligence

Even if ChatGPT itself is, systems built on top are definitely not, this is just getting starting.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#137

Post by simonw »

gwright wrote: post #132
ChatGTP wrote: post #101

Earlier quoted context omitted.

So do you find it alarming that people are trying to give such a system “agency” ?

I do. I think the anthropomorphic language that people use to describe these systems is inaccurate and misleading. An Australian mayor has claimed that ChatGPT "defamed" him. The title of this article says that we should teach people that text generation tools "lie". Other articles suggest that ChatGPT "knows" things. It is extremely interesting to me how much milage can be gotten out of an LLM by observing patterns…

I agree with you: convincing people that these systems do not have intelligence or agency is the second most important problem to solve.

The most important problem is to ensure people understand that these systems cannot be trusted to tell them the truth.

I'm OK with starting out with "these systems will lie to you", then following up with "but you do need to understand that they're not anthropomorphic etc" later on.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#138

Post by Kranar »

jki275 wrote: post #124
Kranar wrote: post #79

Earlier quoted context omitted.

You could also just ask it: "What years did the Mets when the NL East division?" Without any reference to anything and it will give you the correct answer as well. You have been duped into believing that ChatGPT reads websites. It doesn't.

You missed what he said. He copied the actual data into chatgpt as part of the prompt, and it gave the correct information. without that data, it might, or might not, give correct info, and often it won't.

You can't feed anything more than a fairly short Wikipedia article into ChatGPT, its context window isn't remotely close to big enough to do that.

It also doesn't change the point that copying data has no effect. You could just ask ChatGPT what years the Mets won and it will tell you the correct answer.

To test this, I pasted the Wikipedia information but I changed the data, I just gave ChatGPT incorrect information about the Mets, and then I asked it the same question.

ChatGPT ignored the data I provided it and instead gave me the correct information, so what I pasted had no effect on its output.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#139

Post by DebtDeflation »

zmmmmm wrote: post #52

I think characterisation of LLMs as lying is reasonable because although the intent isn't there to misrepresent the truth in answering the specific query, the intent is absolutely there in how the network is trained. The training algorithm is designed to create the most plausible text possible - decoupled from the truthfulness of the output. In a lot of cases (indeed most cases) the easiest way to make the text plaus…

IMO, it just requires the same level of skepticism as a Google search. Just because you enter a query into the search bar and Google returns a list of links and you click one of those links and it contains content that makes a claim, doesn't mean that claim is correct. After all, this is largely what GPT has been trained on.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#140

Post by sltkr »

chaxor wrote: post #84
TuringTest wrote: post #66

Large language models have read everything, and they don't know anything. They are excellent imitators, being able to clone the style and contents of any subject or source you ask for. When you prompt them, they will uncritically generate a text that combines the relevant topics in creative ways, without the least understanding of their meaning. Their original training causes them to memorize lots of concepts , both…

Can you prove that it actually "doesn't know anything"? What do you mean by that? Being critical does not make you educated on the subject. There are so many comments like this, yet never provide any useful information. Saying there's no value in something, as everyone seems to try to do regarding LLMs, should come with more novel insights than parroting this same idea along with every single person on HN.

Especially ironic that the original comment came from a user named "TuringTest"!
Post reply on HN