Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT produces made-up nonexistent references

news.ycombinator.com

Post reply on HN
Print view View original

141–150 of 162 posts

Re: ChatGPT produces made-up nonexistent references

#141

Post by skybrian »

gtsnexp wrote: post #65

I got a very different output for the exact same prompt: "I'm sorry, but I am a large language model trained by OpenAI and do not have access to academic references or other external information. My knowledge is based on the text that I was trained on, which has a fixed cutoff point in time. I do not have the ability to browse the internet or access new information that may have been published since my training data…

It's not enough to use the same prompt unless that was the first thing they asked, because the chat history gets fed back in.

Re: ChatGPT produces made-up nonexistent references

#142

Post by xiphias2 »

wcoenen wrote: post #53
doomlaser wrote: post #30

This is called "hallucination" and I find it to be the biggest flaw with GPT-3 interfaces like ChatGPT. Basically, the system will start convincingly BSing at any point and as a user, you're never sure if any particular detail it outputs is actually correct. The problem is, these large language models are inherently kind of a black box, so how do you fix this kind of behavior? I started looking and it turns out that…

I'm not sure "hallucination" is the right word. I've seen it referred to as "stochastic parroting" elsewhere, and that probably gives more insight into what is happening. These large language models are trained to predict the next word for a given input. And they don't have a choice about this; they must predict the next word, even if it means that they have to make something up. So perhaps the solution would be to i…

,,The word "hallucination" itself was introduced into the English language by the 17th-century physician Sir Thomas Browne in 1646 from the derivation of the Latin word alucinari meaning to wander in the mind. For Browne, hallucination means a sort of vision that is "depraved and receive[s] its objects erroneously".[8]''

I'm not sure if we know enough about hallucination to confirm that it's that much different from what GPT is doing.

Re: ChatGPT produces made-up nonexistent references

#143

Post by lxgr »

doomlaser wrote: post #30

This is called "hallucination" and I find it to be the biggest flaw with GPT-3 interfaces like ChatGPT. Basically, the system will start convincingly BSing at any point and as a user, you're never sure if any particular detail it outputs is actually correct. The problem is, these large language models are inherently kind of a black box, so how do you fix this kind of behavior? I started looking and it turns out that…

> Basically, the system will start convincingly BSing at any point and as a user, you're never sure if any particular detail it outputs is actually correct.

That almost sounds like it was trained on first level customer support chats/calls.

Re: ChatGPT produces made-up nonexistent references

#144

Post by version_five »

trilbyglens wrote: post #86
wcoenen wrote: post #53

Earlier quoted context omitted.

I'm not sure "hallucination" is the right word. I've seen it referred to as "stochastic parroting" elsewhere, and that probably gives more insight into what is happening. These large language models are trained to predict the next word for a given input. And they don't have a choice about this; they must predict the next word, even if it means that they have to make something up. So perhaps the solution would be to i…

Isn't the problem more _because_ it's a language model, and not a knowledge model? It's not based on facts, or even able to go out and find facts. If it's not in the training set it simply doesn't know. It seems like this is only a single layer to something that should be larger. It should be able to tell if what it's saying is true, or to go out and find facts when it's missing them. The fact that it's only a langua…

> It seems like this is only a single layer to something that should be larger

Absolutely correct, and I believe anyone working on these models would agree and, other than as a fun demo, would never suggest that the raw model output gets used for any real purpose. A similar analogy would be self-driving cars. Somewhere "under the hood" there is an ML computer vision model, but it's not like the output layer is just hooked up to the gas and steering. There is all sorts of other logic to make sure the car behaves as intended and fails gracefully under ambiguity.

People see these language models and their flaws and somehow interpret it as a flawed overall product, when they are instead just seeing the underlying model. Admittedly, openAI hasn't helped much by building and promoting a chatbot the way they have.

Lots of cool potential for large language models, very little that comes from raw interaction

Re: ChatGPT produces made-up nonexistent references

#146

Post by dr_zoidberg »

ghaff wrote: post #117
Al-Khwarizmi wrote: post #115

Earlier quoted context omitted.

Guess what, actual school essays also tend to have a bunch of errors and poor reasoning... especially those by kids who would be inclined to cheat rather than write their own essay. Sure, you can nitpick for errors, but do you really think something like this wouldn't cut it at school? We are not talking about writing technical papers or laws, but about writing at a relatively novice level... > Write a school essay o…

At the high school level especially? I just asked it to write an essay on the Battle of Agincourt. Perfectly serviceable prose that looks accurate to me. I'm sure I could flesh it out using some follow-on questions. What were the causes? Describe the reign of Henry V. Describe the English tactics and why they won. Etc. Add a few convincing references to any of the doubtless many books on the subject for sale on Amazo…

I'm answering both you and the previous one in the chain. I used huggingface's OpenAI detector[0], the pizza example is detected as fake with 99.98% probability, whereas the Battle of Agincourt text is also reported as fake with over 99% probability. I don't have any example at hand of how much "real" text on these topics to try and see what that model detects, but up to this point[1] this text is reported as 99% real. You can play and see how much % it assigns to different examples.[2]

[0] https://huggingface.co/openai-detector

[1] I had to cut somewhere to copy and paste into the detector, so [1] is the cutting point

[2] My whole response was still detected as 99.98% real, and copy-pasting the Agincourt bit at the end still reported as 99.98% real. However, moving it to the start (having my answer as a last paragraph) made it detect as 97.71% real. Deleting only "huggingface's" balanced it as 54% real. You can play around with the demo to test different variations of any text you analyze with it.

Re: ChatGPT produces made-up nonexistent references

#147

Post by bjourne »

wcoenen wrote: post #53
doomlaser wrote: post #30

This is called "hallucination" and I find it to be the biggest flaw with GPT-3 interfaces like ChatGPT. Basically, the system will start convincingly BSing at any point and as a user, you're never sure if any particular detail it outputs is actually correct. The problem is, these large language models are inherently kind of a black box, so how do you fix this kind of behavior? I started looking and it turns out that…

I'm not sure "hallucination" is the right word. I've seen it referred to as "stochastic parroting" elsewhere, and that probably gives more insight into what is happening. These large language models are trained to predict the next word for a given input. And they don't have a choice about this; they must predict the next word, even if it means that they have to make something up. So perhaps the solution would be to i…

Look at the titles, author names, and links. The model would be very confident in that output because it is so close to the mean. The model doesn't know that it is confused, instead it confidently parrots the most generic, bland continuation in can come up with.

Re: ChatGPT produces made-up nonexistent references

#148

Post by ghaff »

dr_zoidberg wrote: post #146
ghaff wrote: post #117

Earlier quoted context omitted.

At the high school level especially? I just asked it to write an essay on the Battle of Agincourt. Perfectly serviceable prose that looks accurate to me. I'm sure I could flesh it out using some follow-on questions. What were the causes? Describe the reign of Henry V. Describe the English tactics and why they won. Etc. Add a few convincing references to any of the doubtless many books on the subject for sale on Amazo…

I'm answering both you and the previous one in the chain. I used huggingface's OpenAI detector[0], the pizza example is detected as fake with 99.98% probability, whereas the Battle of Agincourt text is also reported as fake with over 99% probability. I don't have any example at hand of how much "real" text on these topics to try and see what that model detects, but up to this point[1] this text is reported as 99% rea…

You play with ChatGPT a bit and you do start to see certain patterns in the results. So I guess if you train on the outputs it gets relatively straightforward to recognize other examples that are just cut and pasted from the ChatGPT--even if, in isolation, it seems like something a person might write.

Re: ChatGPT produces made-up nonexistent references

#149

Post by zweifuss »

layer8 wrote: post #106
ghaff wrote: post #99

Earlier quoted context omitted.

I was playing around with ChatGPT yesterday and what I found was: - Ask a straightforward question like what are business models for open source software or what are the advantages of GPL vs. AGPL vs. MIT and the answers were pretty shallow and boilerplate but they were correct as far as they went. - But ask a question like "Why didn't large SMP systems become more mainstream?" (which basically requires synthesizing…

Also, if you’d ask it “Why did large SMP systems become mainstream?", it would (presumably) spit out a plausible-sounding explanation for that. The models seem to take the prompt as a premise, and then go from there. Maybe that’s also why when you point out a mistake, they oblige and provide a correction. I haven’t tried: If you point out something they said as a mistake when in fact it was correct, do they notice an…

I did ask both questions and found the answer to "Why did large SMP systems become mainstream?" more coherent. Broken down into bullet points, the arguments given for both questions would be AFAIK correct.
Post reply on HN