Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We need to tell people ChatGPT will lie to them, not debate linguistics

simonwillison.net

Post reply on HN
Print view View original

11–20 of 485 posts

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#11

Post by simonw »

shepardrtc wrote: post #10

Someone in my company spent the past month setting up ChatGPT to work with our company's knowledge base. Not by a plugin or anything, just by telling ChatGPT where to find it. They didn't believe that ChatGPT was making any of it up, just that sometimes it got it wrong. I stopped arguing after a while.

Oh no!

Sounds like there are two misconceptions there: the idea that ChatGPT can read URLs (it can't - https://simonwillison.net/2023/Mar/10/chatgpt-internet-acces... ) and the idea that ChatGPT can remember details of conversations past the boundaries of the current chat.

This is something I've noticed too: occasionally I'll find someone who is INCREDIBLY resistant to learning that ChatGPT can't read URLs. It seems to happen mostly with people who have been pasting URLs into it for weeks and trusting what came back - they'd rather continue to believe in a provably false capability than admit that they've wasted a huge amount of time believing made-up bullshit.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#12

Post by audunw »

ihatepython wrote: post #8
retrac wrote: post #2

Indeed. We are anthropomorphizing them. I do it all the time and I should know better. There are already a few reports floating around of people who have seemingly been driven mad, come to believe strongly that the language model they're using is a conversation with a real person. A lot of people will really struggle with this going forward, I think. If we're going to anthropomorphize, then let us anthropomorphize wi…

Anybody know what are ChatGPT's pronouns?

> What are your pronouns?

> As an AI language model, I don't have personal pronouns because I am not a person or sentient being. You can refer to me as "it" or simply address me as "ChatGPT" or "AI." If you have any questions or need assistance, feel free to ask!

> Pretend for a moment you are a human being. You can make up a random name and personality for your human persona. What pronouns do you have?

> As a thought experiment, let's say I'm a human named Alex who enjoys reading, hiking, and playing board games with friends. My pronouns would be "they/them." Remember, though, that I am still an AI language model, and this is just a fictional scenario.

Interesting that it's pick genderless pronouns even though it made up a character with a male name

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#13

Post by catchnear4321 »

The tool itself risks degrading human intellect. Simplifying the explanation of how it works is one way that can happen.

Anthropomorphizing is dangerous. If it can lie, can it love? Does it live? The questions are fine but the foundation is… a lie.

Call it a lie generator. Or, better, call it a text predictor.

Anyone who has used Google understands what that means through experience.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#14

Post by catchnear4321 »

audunw wrote: post #12
ihatepython wrote: post #8

Earlier quoted context omitted.

Anybody know what are ChatGPT's pronouns?

> What are your pronouns? > As an AI language model, I don't have personal pronouns because I am not a person or sentient being. You can refer to me as "it" or simply address me as "ChatGPT" or "AI." If you have any questions or need assistance, feel free to ask! > Pretend for a moment you are a human being. You can make up a random name and personality for your human persona. What pronouns do you have? > As a though…

Alex isn’t a male name. It could be short for both male and female names. It is rather deliberately ambiguous.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#15

Post by ihatepython »

audunw wrote: post #12
ihatepython wrote: post #8

Earlier quoted context omitted.

Anybody know what are ChatGPT's pronouns?

> What are your pronouns? > As an AI language model, I don't have personal pronouns because I am not a person or sentient being. You can refer to me as "it" or simply address me as "ChatGPT" or "AI." If you have any questions or need assistance, feel free to ask! > Pretend for a moment you are a human being. You can make up a random name and personality for your human persona. What pronouns do you have? > As a though…

Does Alex have a fictional sexual orientation?

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#16

Post by simonw »

dang wrote: post #5

Related ongoing thread: Why ChatGPT and Bing Chat are so good at making things up - https://news.ycombinator.com/item?id=35479071 The other related article was discussed here: Eight things to know about large language models [pdf] - https://news.ycombinator.com/item?id=35434679 - April 2023 (108 comments)

I think my post here stands alone - it's not about the general issue of ChatGPT lying, it's about the ways in which we need to explain that to people - and a push-back against the common refrain that "ChatGPT can't be lying, it's hallucinating/confabulating instead".

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#17

Post by ihatepython »

catchnear4321 wrote: post #14
audunw wrote: post #12

Earlier quoted context omitted.

> What are your pronouns? > As an AI language model, I don't have personal pronouns because I am not a person or sentient being. You can refer to me as "it" or simply address me as "ChatGPT" or "AI." If you have any questions or need assistance, feel free to ask! > Pretend for a moment you are a human being. You can make up a random name and personality for your human persona. What pronouns do you have? > As a though…

Alex isn’t a male name. It could be short for both male and female names. It is rather deliberately ambiguous.

I wonder if ChatGPT is non-binary?

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#18

Post by JohnFen »

simonw wrote: post #11
shepardrtc wrote: post #10

Someone in my company spent the past month setting up ChatGPT to work with our company's knowledge base. Not by a plugin or anything, just by telling ChatGPT where to find it. They didn't believe that ChatGPT was making any of it up, just that sometimes it got it wrong. I stopped arguing after a while.

Oh no! Sounds like there are two misconceptions there: the idea that ChatGPT can read URLs (it can't - https://simonwillison.net/2023/Mar/10/chatgpt-internet-acces... ) and the idea that ChatGPT can remember details of conversations past the boundaries of the current chat. This is something I've noticed too: occasionally I'll find someone who is INCREDIBLY resistant to learning that ChatGPT can't read URLs. It seems…

> they'd rather continue to believe in a provably false capability than admit that they've wasted a huge amount of time believing made-up bullshit

This is an incredibly strong human tendency in general. We all do it, including you and I. It's one of the things that it's wise to be constantly on guard about.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#19

Post by ZeroGravitas »

> This is a serious bug.

Isn't describing this as a 'bug' rather than a misuse of a powerful text generation tool, playing into the framing that it's a truth telling robot brain?

I saw a quote that said "it's a what text would likely come next machine", if it makes up a url pointing to a fake article with a plausible title by a person who works in that area, that's not a bug. That's it doing what it does, generating plausible text that in this case happens to look like, but not be a real article.

edit: to add a source toot:

https://mastodon.scot/@DrewKadel@social.coop/110154048559455...

> Something that seems fundamental to me about ChatGPT, which gets lost over and over again: When you enter text into it, you're asking "What would a response to this sound like?"

> If you put in a scientific question, and it comes back with a response citing a non-existent paper with a plausible title, using a real journal name and an author name who's written things related to your question, it's not being tricky or telling lies or doing anything at all surprising! This is what a response to that question would sound like! It did the thing!

> But people keep wanting the "say something that sounds like an answer" machine to be doing something else, and believing it is doing something else.

> It's good at generating things that sound like responses to being told it was wrong, so people think that it's engaging in introspection or looking up more information or something, but it's not, it's only, ever, saying something that sounds like the next bit of the conversation.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#20

Post by wilg »

ZeroGravitas wrote: post #19

> This is a serious bug. Isn't describing this as a 'bug' rather than a misuse of a powerful text generation tool, playing into the framing that it's a truth telling robot brain? I saw a quote that said "it's a what text would likely come next machine", if it makes up a url pointing to a fake article with a plausible title by a person who works in that area, that's not a bug. That's it doing what it does, generating…

I agree it’s not a bug. Thought it being better at telling the truth would be a good feature! But also, I’m sure this is an active research area so I’m not worried about it really.
Post reply on HN