Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT produces made-up nonexistent references

news.ycombinator.com

Post reply on HN
Print view View original

151–160 of 162 posts

Re: ChatGPT produces made-up nonexistent references

#151

Post by dr_zoidberg »

ghaff wrote: post #148
dr_zoidberg wrote: post #146

Earlier quoted context omitted.

I'm answering both you and the previous one in the chain. I used huggingface's OpenAI detector[0], the pizza example is detected as fake with 99.98% probability, whereas the Battle of Agincourt text is also reported as fake with over 99% probability. I don't have any example at hand of how much "real" text on these topics to try and see what that model detects, but up to this point[1] this text is reported as 99% rea…

You play with ChatGPT a bit and you do start to see certain patterns in the results. So I guess if you train on the outputs it gets relatively straightforward to recognize other examples that are just cut and pasted from the ChatGPT--even if, in isolation, it seems like something a person might write.

The "weird" (or "funny" or whatever you want to call it) thing is that the detector I linked was originally made for GPT-2, but still works reasonably well for GPT-3 (and derived) output.

Re: ChatGPT produces made-up nonexistent references

#153

Post by aenis »

shrimpx wrote: post #113
aenis wrote: post #104

Earlier quoted context omitted.

Just ask it what day it is. No detailed scrutiny needed.

I wonder if GPT models can get super precise or if there’s a fundamental limitation.

Indeed. Its a language model, not a knowledge model. But.. I would assume our brains too use language models to provide immediate, lazy responses - and resort to higher level models only when needed :-)

Re: ChatGPT produces made-up nonexistent references

#154

Post by nnlsoccer »

rzzzt wrote: post #66
wcoenen wrote: post #53

Earlier quoted context omitted.

I'm not sure "hallucination" is the right word. I've seen it referred to as "stochastic parroting" elsewhere, and that probably gives more insight into what is happening. These large language models are trained to predict the next word for a given input. And they don't have a choice about this; they must predict the next word, even if it means that they have to make something up. So perhaps the solution would be to i…

If you would like another Latin word for it, take "confabulation" from neuroscience-land: https://en.wikipedia.org/wiki/Confabulation#Signs_and_sympto...

Came here to say the same thing. Medically confabulation is different than hallucination and far more similar to what is being described. Confabulation is seen with wernike-korsokoff syndrome which can be found in very long time alcohol use disorder. The patient makes up stories to fit the gaps in their memory without necessarily realizing that is what they are doing.

Whereas hallucinations are more like present sensory disturbances happening at that moment.

Re: ChatGPT produces made-up nonexistent references

#155

Post by gorbachev »

scotty79 wrote: post #88
doomlaser wrote: post #30

This is called "hallucination" and I find it to be the biggest flaw with GPT-3 interfaces like ChatGPT. Basically, the system will start convincingly BSing at any point and as a user, you're never sure if any particular detail it outputs is actually correct. The problem is, these large language models are inherently kind of a black box, so how do you fix this kind of behavior? I started looking and it turns out that…

I am currently having a lot of fun trying to figure out how some stuff in Rust works by asking GPT. It gives nice, informative answers, however most of them contain small error. When I point it out it happily agrees that I'm correct and helpfully explains why in fact what it told me before was wrong in that detail. This actually might be cool way to learn programming. When you have a compiler at hand to check things…

What happens if you correct it on something that's not wrong or correcting it with another mistake?

Re: ChatGPT produces made-up nonexistent references

#156

Post by ShamelessC »

skybrian wrote: post #141
gtsnexp wrote: post #65

I got a very different output for the exact same prompt: "I'm sorry, but I am a large language model trained by OpenAI and do not have access to academic references or other external information. My knowledge is based on the text that I was trained on, which has a fixed cutoff point in time. I do not have the ability to browse the internet or access new information that may have been published since my training data…

It's not enough to use the same prompt unless that was the first thing they asked, because the chat history gets fed back in.

It shows a more nuanced situation than OP presented. What OP has done is usually called cherrypicking. Ideally you should run the query several times (in more than one context/chat history) and count the responses.

Re: ChatGPT produces made-up nonexistent references

#157

Post by tyra3l »

rzzzt wrote: post #66
wcoenen wrote: post #53

Earlier quoted context omitted.

I'm not sure "hallucination" is the right word. I've seen it referred to as "stochastic parroting" elsewhere, and that probably gives more insight into what is happening. These large language models are trained to predict the next word for a given input. And they don't have a choice about this; they must predict the next word, even if it means that they have to make something up. So perhaps the solution would be to i…

If you would like another Latin word for it, take "confabulation" from neuroscience-land: https://en.wikipedia.org/wiki/Confabulation#Signs_and_sympto...

or simply use the Filling-in: https://en.wikipedia.org/wiki/Filling-in

Re: ChatGPT produces made-up nonexistent references

#158

Post by Al-Khwarizmi »

dr_zoidberg wrote: post #146
ghaff wrote: post #117

Earlier quoted context omitted.

At the high school level especially? I just asked it to write an essay on the Battle of Agincourt. Perfectly serviceable prose that looks accurate to me. I'm sure I could flesh it out using some follow-on questions. What were the causes? Describe the reign of Henry V. Describe the English tactics and why they won. Etc. Add a few convincing references to any of the doubtless many books on the subject for sale on Amazo…

I'm answering both you and the previous one in the chain. I used huggingface's OpenAI detector[0], the pizza example is detected as fake with 99.98% probability, whereas the Battle of Agincourt text is also reported as fake with over 99% probability. I don't have any example at hand of how much "real" text on these topics to try and see what that model detects, but up to this point[1] this text is reported as 99% rea…

Very interesting, I didn't know that detector. But making small local changes, it's easy to make the probability of "fake" go down, while still benefitting from the automatically-written essay. As long as students have access to the same tools as teachers, it will be easy for them to get away with automatically generated essays.

Re: ChatGPT produces made-up nonexistent references

#160

Post by marina_hithere »

Thanks for you post! I spent a few hours verifying each and every academic piece it provided me and they were all false, nonexistent made up resources. I started investigations and googled everything it shared with me. All doctors, researchers and studies were fake, links were fake, insights, study summaries were fake. But looked convincing and real. I am still processing it. We have a long chat about truth and lies and that fake information can be harmful but the model was absolutely convinced it is saying the truth. I called genius evil. I think our online space can be in danger if such systems will be adopted. Humans may make wrong ethical choices or decisions about their money or health. Everybody seem so excited about this chat but there is potential for harm here.
Post reply on HN