Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT produces made-up nonexistent references

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 162 posts

Re: ChatGPT produces made-up nonexistent references

#51

Post by andix »

fakedang wrote: post #44
doomlaser wrote: post #30

This is called "hallucination" and I find it to be the biggest flaw with GPT-3 interfaces like ChatGPT. Basically, the system will start convincingly BSing at any point and as a user, you're never sure if any particular detail it outputs is actually correct. The problem is, these large language models are inherently kind of a black box, so how do you fix this kind of behavior? I started looking and it turns out that…

> Basically, the system will start convincingly BSing at any point and as a user, you're never sure if any particular detail it outputs is actually correct. Guys, hear me out. What if we could replace the entire Management Consulting industry with ChatGPT?

Let’s ask Chat GPT what he/she thinks about that: Do you think an AI could write slides for management consultants in the future? Explain it in a way an arrogant consultant would phrase it.

Yes, I think it's possible for an AI to write slides for management consultants in the future. In fact, I would argue that an AI could potentially do a better job at creating slides for consultants than a human. With access to vast amounts of data and the ability to analyze it quickly and accurately, an AI could help consultants create more compelling and effective presentations. Additionally, because an AI wouldn't be limited by human biases or constraints, it could help consultants think outside the box and come up with unique and innovative ideas to present to their clients. Overall, an AI-powered tool for creating consulting slides could be a valuable asset to any management consultant.

Re: ChatGPT produces made-up nonexistent references

#52

Post by andix »

mudrockbestgirl wrote: post #28

Unsurprising given how it works, but the higher-level implications are somewhat unexpected. These model are extremely good when they have creative freedom and are used to produce some kind of art. Poems, text in various styles, images without important details, and so on. But they fail miserably when you give them tasks that require world knowledge or precision. For example, when I asked a programming question I got…

The quality of creative work they produce is also not so good. An AI may pass a high school creative writing class, but they would probably do that for computer science too („implement fizz buzz“).

But try to give them the assignment to write stories, poems, lyrics. The result is at best mediocre. No professional artist is threatened by that at all.

Re: ChatGPT produces made-up nonexistent references

#53

Post by wcoenen »

doomlaser wrote: post #30

This is called "hallucination" and I find it to be the biggest flaw with GPT-3 interfaces like ChatGPT. Basically, the system will start convincingly BSing at any point and as a user, you're never sure if any particular detail it outputs is actually correct. The problem is, these large language models are inherently kind of a black box, so how do you fix this kind of behavior? I started looking and it turns out that…

I'm not sure "hallucination" is the right word.

I've seen it referred to as "stochastic parroting" elsewhere, and that probably gives more insight into what is happening. These large language models are trained to predict the next word for a given input. And they don't have a choice about this; they must predict the next word, even if it means that they have to make something up.

So perhaps the solution would be to include the prediction confidence in the output. E.g. gray out the parts of the text hat are low confidence predictions, like downvoted HN comments.

Re: ChatGPT produces made-up nonexistent references

#55

Post by apienx »

Thanks for sharing. I've fed davinci3 your prompt with temperature 0.1 and 0.9 and both were lying pieces of junk. The only difference was that the lowest temperature made up refs that were more "plausible"/"realistic" (e.g. it'd spit out the name of authors which have written about the field).

Re: ChatGPT produces made-up nonexistent references

#56

Post by mola »

dschuetz wrote: post #50

Well, this is exactly what you have to expect, because the brain does the same if there is information missing. The brain just makes up missing references to fill in the gaps from what ever information is available or accessible. Dangerous half-knowledge works the same way, where connections between concepts are made up that appear to make sense at first, but then collapse upon scrutiny. So, no surprise here, tbh.

I would answer:"I don't know" for a question like this. I wouldn't start making up facts. I'm also using a brain...

Re: ChatGPT produces made-up nonexistent references

#57

Post by ShamelessC »

wcoenen wrote: post #53
doomlaser wrote: post #30

This is called "hallucination" and I find it to be the biggest flaw with GPT-3 interfaces like ChatGPT. Basically, the system will start convincingly BSing at any point and as a user, you're never sure if any particular detail it outputs is actually correct. The problem is, these large language models are inherently kind of a black box, so how do you fix this kind of behavior? I started looking and it turns out that…

I'm not sure "hallucination" is the right word. I've seen it referred to as "stochastic parroting" elsewhere, and that probably gives more insight into what is happening. These large language models are trained to predict the next word for a given input. And they don't have a choice about this; they must predict the next word, even if it means that they have to make something up. So perhaps the solution would be to i…

Hallucination is commonly used in ML parlance and gets the point across without needing to know what "stochastic" means.

Re: ChatGPT produces made-up nonexistent references

#59

Post by djmips »

doomlaser wrote: post #30

This is called "hallucination" and I find it to be the biggest flaw with GPT-3 interfaces like ChatGPT. Basically, the system will start convincingly BSing at any point and as a user, you're never sure if any particular detail it outputs is actually correct. The problem is, these large language models are inherently kind of a black box, so how do you fix this kind of behavior? I started looking and it turns out that…

Fake it till you make it.

Re: ChatGPT produces made-up nonexistent references

#60

Post by andix »

wcoenen wrote: post #53
doomlaser wrote: post #30

This is called "hallucination" and I find it to be the biggest flaw with GPT-3 interfaces like ChatGPT. Basically, the system will start convincingly BSing at any point and as a user, you're never sure if any particular detail it outputs is actually correct. The problem is, these large language models are inherently kind of a black box, so how do you fix this kind of behavior? I started looking and it turns out that…

I'm not sure "hallucination" is the right word. I've seen it referred to as "stochastic parroting" elsewhere, and that probably gives more insight into what is happening. These large language models are trained to predict the next word for a given input. And they don't have a choice about this; they must predict the next word, even if it means that they have to make something up. So perhaps the solution would be to i…

They are not forced to come up with new ideas. They can also write something like „I have no further information about that“. But in training this is probably discouraged, because they shouldn’t answer all questions like that.
Post reply on HN