Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We need to tell people ChatGPT will lie to them, not debate linguistics

simonwillison.net

Post reply on HN
Print view View original

31–40 of 485 posts

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#31

Post by lm28469 »

zug_zug wrote: post #27
retrac wrote: post #2

Indeed. We are anthropomorphizing them. I do it all the time and I should know better. There are already a few reports floating around of people who have seemingly been driven mad, come to believe strongly that the language model they're using is a conversation with a real person. A lot of people will really struggle with this going forward, I think. If we're going to anthropomorphize, then let us anthropomorphize wi…

That all feels tad dramatic. It's like a person on the internet -- in that it's wrong 20% of the time, often confidently so. But the distinction is it's less rude, and more knowledgeable.

> But the distinction is it's less rude, and more knowledgeable.

And that regular people assume it basically is an oracle, which doesn't happen to many people online

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#32

Post by lm28469 »

simonw wrote: post #11
shepardrtc wrote: post #10

Someone in my company spent the past month setting up ChatGPT to work with our company's knowledge base. Not by a plugin or anything, just by telling ChatGPT where to find it. They didn't believe that ChatGPT was making any of it up, just that sometimes it got it wrong. I stopped arguing after a while.

Oh no! Sounds like there are two misconceptions there: the idea that ChatGPT can read URLs (it can't - https://simonwillison.net/2023/Mar/10/chatgpt-internet-acces... ) and the idea that ChatGPT can remember details of conversations past the boundaries of the current chat. This is something I've noticed too: occasionally I'll find someone who is INCREDIBLY resistant to learning that ChatGPT can't read URLs. It seems…

> they'd rather continue to believe in a provably false capability

For some people it borderline is a cult now, even here on HN, they'll give it intelligence, personality, character, &c.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#33

Post by lcnPylGDnU4H9OF »

travisjungroth wrote: post #28
ZeroGravitas wrote: post #19

> This is a serious bug. Isn't describing this as a 'bug' rather than a misuse of a powerful text generation tool, playing into the framing that it's a truth telling robot brain? I saw a quote that said "it's a what text would likely come next machine", if it makes up a url pointing to a fake article with a plausible title by a person who works in that area, that's not a bug. That's it doing what it does, generating…

I like the definition of bug as “unexpected behavior”. So this isn’t a bug when it comes the underlying service. But for ChatGPT, a consumer-facing web app that can “answer followup questions, admit its mistakes, challenge false premises and reject inappropriate requests”, then making stuff up and passing it off as true is unexpected behavior.

It sounds like this is unexpected behavior, even from the perspective of those developing at the lowest level in these models.

From the essay:

> What I find fascinating about this is that these extremely problematic behaviours are not the system working as intended: they are bugs! And we haven’t yet found a reliable way to fix them.

Right below that is this link: https://arxiv.org/abs/2212.09251. From the introduction on that page:

> As language models (LMs) scale, they develop many novel behaviors, good and bad, exacerbating the need to evaluate how they behave.

Especially with a definition as broad as "unexpected behavior", these "novel behaviors" seem to fit. But even without that:

> We also find some of the first examples of inverse scaling in RL from Human Feedback (RLHF), where more RLHF makes LMs worse. For example, RLHF makes LMs express stronger political views ... and a greater desire to avoid shut down.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#35

Post by travisjungroth »

gWPVhyxPHqvk wrote: post #29

After a few weeks playing around with ChatGPT, my workflow with it has become: 0. Try to sign in, see the system is over capacity, leave. Maybe I’ll try again in 10 minutes. 1. Ask my question, get an answer. I’ll have no idea if what I got is real or not. 2. Google for the answer, since I can’t trust the answer 3. Realize I wasted 20 minutes trying to converse with a computer, and resolve that next time I’ll just ty…

If you expect ChatGPT to give you information or direct you to it (like Wikipedia or Google) you will be frequently disappointed. You may also be frequently pleased, but you often won’t be sure which and that’s a problem.

ChatGPT is very good at transforming information. You need to show up with stuff and then have it change that stuff for you somehow. You will be disappointed less often.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#36

Post by revolvingocelot »

zug_zug wrote: post #27
retrac wrote: post #2

Indeed. We are anthropomorphizing them. I do it all the time and I should know better. There are already a few reports floating around of people who have seemingly been driven mad, come to believe strongly that the language model they're using is a conversation with a real person. A lot of people will really struggle with this going forward, I think. If we're going to anthropomorphize, then let us anthropomorphize wi…

That all feels tad dramatic. It's like a person on the internet -- in that it's wrong 20% of the time, often confidently so. But the distinction is it's less rude, and more knowledgeable.

When someone is wrong on the internet, nailing both the tone and vocabulary of the type of expert said wrong someone is purporting to be is rare and impressive. But ChatGPT nails both an overwhelming amount of the time, IME, and in that way it is entirely unlike a person on the internet.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#37

Post by WorldMaker »

retrac wrote: post #2

Indeed. We are anthropomorphizing them. I do it all the time and I should know better. There are already a few reports floating around of people who have seemingly been driven mad, come to believe strongly that the language model they're using is a conversation with a real person. A lot of people will really struggle with this going forward, I think. If we're going to anthropomorphize, then let us anthropomorphize wi…

I think people see the patient/well-read in the text as it reads, but have a harder time distinguishing the other more pyschopathic/delusional tendencies. People don't take some of the precautions because they don't read some of the warning signs (until it is too late).

I keep wondering if it would be useful to add required "teenage" quirks to the output: more filler words like "um" and "like" (maybe even full "Valley Girl" with it?), less "self-assured" vocabulary and more hedges like "I think" and "I read" and "Something I found but I'm not sure about" type things. Less punctuation, more uncaring spelling mistakes.

I don't think we can stop anthropomorphizing them, but maybe we can force training deeper in directions of tics and mannerisms that better flag ahead of time the output is a best-guess approximation from "someone" a bit unreliable. It will probably make them slightly worse as assistants, but slightly better at seeming to be what they are and maybe more people will take precautions in that case.

Maybe we have to force that industry-wide. Force things like ChatGPT to "sound" more like the psychopaths they are so that people more easily take them with a grain of salt, less easily trust them.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#38

Post by simonw »

gWPVhyxPHqvk wrote: post #29

After a few weeks playing around with ChatGPT, my workflow with it has become: 0. Try to sign in, see the system is over capacity, leave. Maybe I’ll try again in 10 minutes. 1. Ask my question, get an answer. I’ll have no idea if what I got is real or not. 2. Google for the answer, since I can’t trust the answer 3. Realize I wasted 20 minutes trying to converse with a computer, and resolve that next time I’ll just ty…

I pay for ChatGPT Plus, and use it with no delays at all dozens of times a day. The more I use it the better I get at predicting if it can be useful for a specific question or not.

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#39

Post by usrusr »

I wonder how the impact on the legal field will eventually turn out: from a certain perspective, that already looks like a battle more of quantity than of quality, throw tons of binders full of hopelessly weak arguments at the other side and if they fail to find the few somewhat sound needles in that haystack they won't be able to prepare a defense. Now enter a tool that can be used to write seemingly infinite amounts of trash legalese the other side has to spend lots of resources on discarding. Will we perhaps see an "unsupported" category of legal procedure, where both sides agree to meet in an offline-only arena?

Re: We need to tell people ChatGPT will lie to them, not debate linguistics

#40

Post by LargeTomato »

retrac wrote: post #2

Indeed. We are anthropomorphizing them. I do it all the time and I should know better. There are already a few reports floating around of people who have seemingly been driven mad, come to believe strongly that the language model they're using is a conversation with a real person. A lot of people will really struggle with this going forward, I think. If we're going to anthropomorphize, then let us anthropomorphize wi…

>like having an assistant who is patient, incredibly well-read, sycophantic, impressionable, amoral, psychopathic, and prone to bouts of delusional confidence and confabulation.

So basically an assistant with bipolar disorder.

I have BP. At various times I can be all of those things, although perhaps not so much a psychopath.

Post reply on HN