Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT produces made-up nonexistent references

news.ycombinator.com

Post reply on HN
Print view View original

131–140 of 162 posts

Re: ChatGPT produces made-up nonexistent references

#131

Post by [deleted] »

DiogenesKynikos wrote: post #129
mudrockbestgirl wrote: post #28

Unsurprising given how it works, but the higher-level implications are somewhat unexpected. These model are extremely good when they have creative freedom and are used to produce some kind of art. Poems, text in various styles, images without important details, and so on. But they fail miserably when you give them tasks that require world knowledge or precision. For example, when I asked a programming question I got…

> Poems, text in various styles, images without important details, and so on. But they fail miserably when you give them tasks that require world knowledge or precision. So what you're telling me is that STEM is harder than humanities?

[deleted]

Re: ChatGPT produces made-up nonexistent references

#132

Post by theptip »

Google’s model LaMDA has a system for detecting and avoiding false info, it’s interesting:

https://arxiv.org/pdf/2201.08239.pdf

Basically they have a fact-checker layer that calls the internet and replaces incorrect facts in the model’s output.

IOW this is a hard problem and GPT isn’t engineered to avoid it.

Re: ChatGPT produces made-up nonexistent references

#133

Post by layer8 »

ghaff wrote: post #110
layer8 wrote: post #106

Earlier quoted context omitted.

Also, if you’d ask it “Why did large SMP systems become mainstream?", it would (presumably) spit out a plausible-sounding explanation for that. The models seem to take the prompt as a premise, and then go from there. Maybe that’s also why when you point out a mistake, they oblige and provide a correction. I haven’t tried: If you point out something they said as a mistake when in fact it was correct, do they notice an…

Interesting. I'll have to try that. And, yes, you are correct. It gave a short pablum answer that pretty much focused on the benefits of multiple cores. The mistake in the original was somewhat subtle and gets into programming for large SMP vs. MPP/distributed systems. And it's actually true that parallel programming even on shared memory systems has challenges (and there was a lot of discussion at one point when mul…

See https://news.ycombinator.com/item?id=33844574 where the model acknowledges a non-existing mistake after prompting.

Re: ChatGPT produces made-up nonexistent references

#134

Post by rcthompson »

wcoenen wrote: post #53
doomlaser wrote: post #30

This is called "hallucination" and I find it to be the biggest flaw with GPT-3 interfaces like ChatGPT. Basically, the system will start convincingly BSing at any point and as a user, you're never sure if any particular detail it outputs is actually correct. The problem is, these large language models are inherently kind of a black box, so how do you fix this kind of behavior? I started looking and it turns out that…

I'm not sure "hallucination" is the right word. I've seen it referred to as "stochastic parroting" elsewhere, and that probably gives more insight into what is happening. These large language models are trained to predict the next word for a given input. And they don't have a choice about this; they must predict the next word, even if it means that they have to make something up. So perhaps the solution would be to i…

I wonder if "cargo culting" would be an accurate characterization.

Re: ChatGPT produces made-up nonexistent references

#135

Post by zdimension »

Some of the links, funnily, are based on "real" links: https://www.cs.cmu.edu/~fp/courses/lp/lectures.html is broken but https://www.cs.cmu.edu/~fp/courses/lp/ is the page for a Logic Programming course.

https://www.cs.purdue.edu/homes/hosking/726/html/prolog_list... is broken, but there are many pages in the format https://www.cs.purdue.edu/homes/hosking/XXX corresponding to courses taught by that guy, but 726 is nonexistent.

Re: ChatGPT produces made-up nonexistent references

#136

Post by Terretta »

dsr_ wrote: post #89
ShamelessC wrote: post #57

Earlier quoted context omitted.

Hallucination is commonly used in ML parlance and gets the point across without needing to know what "stochastic" means.

"Stochastic" means "random, not supported by facts, hypothetical" in every context in which it is used, across many fields. The real problem is that anyone thought that they could pull factual material out of a giant language correlation network.

stochastic screening in printing (as opposed to halftoning) samples constrained random color points from a real/actual image

the uses of stochastic i've seen 'in the wild' have nothing to do with 2/3 of that definition

Re: ChatGPT produces made-up nonexistent references

#137

Post by canjobear »

wcoenen wrote: post #53
doomlaser wrote: post #30

This is called "hallucination" and I find it to be the biggest flaw with GPT-3 interfaces like ChatGPT. Basically, the system will start convincingly BSing at any point and as a user, you're never sure if any particular detail it outputs is actually correct. The problem is, these large language models are inherently kind of a black box, so how do you fix this kind of behavior? I started looking and it turns out that…

I'm not sure "hallucination" is the right word. I've seen it referred to as "stochastic parroting" elsewhere, and that probably gives more insight into what is happening. These large language models are trained to predict the next word for a given input. And they don't have a choice about this; they must predict the next word, even if it means that they have to make something up. So perhaps the solution would be to i…

> And they don't have a choice about this; they must predict the next word, even if it means that they have to make something up.

No, they could easily generate the end-of-sequence symbol, or the words “I don’t know.”

Re: ChatGPT produces made-up nonexistent references

#138

Post by skybrian »

andix wrote: post #60
wcoenen wrote: post #53

Earlier quoted context omitted.

I'm not sure "hallucination" is the right word. I've seen it referred to as "stochastic parroting" elsewhere, and that probably gives more insight into what is happening. These large language models are trained to predict the next word for a given input. And they don't have a choice about this; they must predict the next word, even if it means that they have to make something up. So perhaps the solution would be to i…

They are not forced to come up with new ideas. They can also write something like „I have no further information about that“. But in training this is probably discouraged, because they shouldn’t answer all questions like that.

Yes, the training doesn't encourage this. It encourages guessing, because if it guesses the next word and it's right, the guessing is reinforced.

Whenever the model gets something right, it's the result of good guesses that were reinforced. It's all guesswork, it's just that some guesses are right.

Re: ChatGPT produces made-up nonexistent references

#139

Post by freshnode »

I'm not sure whether this is another example of convincing hallucination but the linked Tweet discussing GPT3 on HN yesterday [1] suggested that adding "browsing: enabled" to the prompt allowed the model to query web pages... Could be worth a try with your prompt?

[1] - https://twitter.com/carnage4life/status/1598332648723976193

Re: ChatGPT produces made-up nonexistent references

#140

Post by FireInsight »

freshnode wrote: post #139

I'm not sure whether this is another example of convincing hallucination but the linked Tweet discussing GPT3 on HN yesterday [1] suggested that adding "browsing: enabled" to the prompt allowed the model to query web pages... Could be worth a try with your prompt? [1] - https://twitter.com/carnage4life/status/1598332648723976193

I tried browsing: enabled today and it always just told me it can't browse the internet.
Post reply on HN