Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We need to tell people ChatGPT will lie to them, not debate linguistics

simonwillison.net

Post reply on HN
Print view View original

21–30 of 485 posts

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#21

Post by wilg »

simonw wrote: post #11
shepardrtc wrote: post #10

Someone in my company spent the past month setting up ChatGPT to work with our company's knowledge base. Not by a plugin or anything, just by telling ChatGPT where to find it. They didn't believe that ChatGPT was making any of it up, just that sometimes it got it wrong. I stopped arguing after a while.

Oh no! Sounds like there are two misconceptions there: the idea that ChatGPT can read URLs (it can't - https://simonwillison.net/2023/Mar/10/chatgpt-internet-acces... ) and the idea that ChatGPT can remember details of conversations past the boundaries of the current chat. This is something I've noticed too: occasionally I'll find someone who is INCREDIBLY resistant to learning that ChatGPT can't read URLs. It seems…

They definitely need to add some guardrails to warn it can’t read URLs like they do when you try to ask anything fun.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#22

Post by simonw »

ZeroGravitas wrote: post #19

> This is a serious bug. Isn't describing this as a 'bug' rather than a misuse of a powerful text generation tool, playing into the framing that it's a truth telling robot brain? I saw a quote that said "it's a what text would likely come next machine", if it makes up a url pointing to a fake article with a plausible title by a person who works in that area, that's not a bug. That's it doing what it does, generating…

I think it's an entire category of bugs.

The thing where you paste in a URL and it says "here is a summary of the content of that page: ..." is very definitely a bug. It's a user experience bug - the system should not confuse people by indicating it can do something that it cannot.

The thing where you ask for a biography of a living person and it throws in 80% real facts and 20% wild hallucinations - like saying they worked for a company that they did not work for - is a bug.

The thing where you ask it for citations and it invents convincing names for academic papers and made-up links to pages that don't exist? That's another bug.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#23

Post by MandieD »

simonw wrote: post #11
shepardrtc wrote: post #10

Someone in my company spent the past month setting up ChatGPT to work with our company's knowledge base. Not by a plugin or anything, just by telling ChatGPT where to find it. They didn't believe that ChatGPT was making any of it up, just that sometimes it got it wrong. I stopped arguing after a while.

Oh no! Sounds like there are two misconceptions there: the idea that ChatGPT can read URLs (it can't - https://simonwillison.net/2023/Mar/10/chatgpt-internet-acces... ) and the idea that ChatGPT can remember details of conversations past the boundaries of the current chat. This is something I've noticed too: occasionally I'll find someone who is INCREDIBLY resistant to learning that ChatGPT can't read URLs. It seems…

Ah, that’s why it messed up my rather simple request to generate a SQL query to get all messages received by a given call sign, based on the table definition at https://wspr.live - it picked plausible, but nonexistent table and column names.

I took “this thing isn’t nearly as smart as everyone’s making it out to be” from that session, but you’re the first person to make it clear that it’s not actually reading the rather simple page I directed it to.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#24

Post by kelseyfrog »

ihatepython wrote: post #17
catchnear4321 wrote: post #14

Earlier quoted context omitted.

Alex isn’t a male name. It could be short for both male and female names. It is rather deliberately ambiguous.

I wonder if ChatGPT is non-binary?

It's evidently clear that ChatGPT is binary. It runs on computer hardware.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#25

Post by shepardrtc »

simonw wrote: post #11
shepardrtc wrote: post #10

Someone in my company spent the past month setting up ChatGPT to work with our company's knowledge base. Not by a plugin or anything, just by telling ChatGPT where to find it. They didn't believe that ChatGPT was making any of it up, just that sometimes it got it wrong. I stopped arguing after a while.

Oh no! Sounds like there are two misconceptions there: the idea that ChatGPT can read URLs (it can't - https://simonwillison.net/2023/Mar/10/chatgpt-internet-acces... ) and the idea that ChatGPT can remember details of conversations past the boundaries of the current chat. This is something I've noticed too: occasionally I'll find someone who is INCREDIBLY resistant to learning that ChatGPT can't read URLs. It seems…

What really tricked them was that our company's website is in its corpus. At least the old version. So when asked about different aspects of our products or company, it would give answers that were close, but not quite correct.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#26

Post by kurthr »

retrac wrote: post #2

Indeed. We are anthropomorphizing them. I do it all the time and I should know better. There are already a few reports floating around of people who have seemingly been driven mad, come to believe strongly that the language model they're using is a conversation with a real person. A lot of people will really struggle with this going forward, I think. If we're going to anthropomorphize, then let us anthropomorphize wi…

Maybe when the AI uprising occurs knowing that they lie and cheat will provide some small consolation?

I'm not really serious, but having watched each generation develop psychological immunity to distracting media/techology (and discussing the impact of radio with those older than myself) it seems like this knowledge could help shield the next generation from some of the negative effects of these new tools.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#27

Post by zug_zug »

retrac wrote: post #2

Indeed. We are anthropomorphizing them. I do it all the time and I should know better. There are already a few reports floating around of people who have seemingly been driven mad, come to believe strongly that the language model they're using is a conversation with a real person. A lot of people will really struggle with this going forward, I think. If we're going to anthropomorphize, then let us anthropomorphize wi…

That all feels tad dramatic.

It's like a person on the internet -- in that it's wrong 20% of the time, often confidently so. But the distinction is it's less rude, and more knowledgeable.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#28

Post by travisjungroth »

ZeroGravitas wrote: post #19

> This is a serious bug. Isn't describing this as a 'bug' rather than a misuse of a powerful text generation tool, playing into the framing that it's a truth telling robot brain? I saw a quote that said "it's a what text would likely come next machine", if it makes up a url pointing to a fake article with a plausible title by a person who works in that area, that's not a bug. That's it doing what it does, generating…

I like the definition of bug as “unexpected behavior”. So this isn’t a bug when it comes the underlying service. But for ChatGPT, a consumer-facing web app that can “answer followup questions, admit its mistakes, challenge false premises and reject inappropriate requests”, then making stuff up and passing it off as true is unexpected behavior.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#29

Post by gWPVhyxPHqvk »

After a few weeks playing around with ChatGPT, my workflow with it has become:

0. Try to sign in, see the system is over capacity, leave. Maybe I’ll try again in 10 minutes.

1. Ask my question, get an answer. I’ll have no idea if what I got is real or not.

2. Google for the answer, since I can’t trust the answer

3. Realize I wasted 20 minutes trying to converse with a computer, and resolve that next time I’ll just type 3 words into Google.

As amazing as the GPTs are, the speed and ease of Google is still unmatched for 95% of knowledge lookup tasks.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#30

Post by [deleted] »

bsaul wrote: post #4
retrac wrote: post #2

Indeed. We are anthropomorphizing them. I do it all the time and I should know better. There are already a few reports floating around of people who have seemingly been driven mad, come to believe strongly that the language model they're using is a conversation with a real person. A lot of people will really struggle with this going forward, I think. If we're going to anthropomorphize, then let us anthropomorphize wi…

I love this description. The only thing left is to generate the corresponding avatar with dall-E

[deleted]
Post reply on HN