Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT produces made-up nonexistent references

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 162 posts

Re: ChatGPT produces made-up nonexistent references

#81

Post by ghayes »

rcfox wrote: post #69
asimpletune wrote: post #67

Earlier quoted context omitted.

I don't think it works that way. The models don't have a database of facts, so they never reach a point where they know that something they're saying is based on the real world. I think in other words, they literally operate by just predicting what comes next and sometimes that stuff is just made up.

ChatGPT has responded to a lot of my requests with an answer along the lines of "I don't have information about that" or "It's impossible to answer that without more information, which I can't get." Sometimes, starting a new session will get it to give an actual answer. Sometimes asking for an estimate or approximation works.

This is covered in ChatGPT’s learn more section:

> Limitations

> ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers. Fixing this issue is challenging, as: (1) during RL training, there’s currently no source of truth; (2) training the model to be more cautious causes it to decline questions that it can answer correctly; and (3) supervised training misleads the model because the ideal answer depends on what the model knows, rather than what the human demonstrator knows.

https://openai.com/blog/chatgpt/

Re: ChatGPT produces made-up nonexistent references

#83

Post by saimiam »

mola wrote: post #56
dschuetz wrote: post #50

Well, this is exactly what you have to expect, because the brain does the same if there is information missing. The brain just makes up missing references to fill in the gaps from what ever information is available or accessible. Dangerous half-knowledge works the same way, where connections between concepts are made up that appear to make sense at first, but then collapse upon scrutiny. So, no surprise here, tbh.

I would answer:"I don't know" for a question like this. I wouldn't start making up facts. I'm also using a brain...

That’s your conscious brain.

People generate false memories, experience phantom pain, see blue or gold depending on the angle of an outfit, and other such brain blips because of the brain making up a reality to fill in gaps. In that light, GPT-3 is behaving much like a human brain.

Re: ChatGPT produces made-up nonexistent references

#84

Post by smileybarry »

This is kind of a funny quirk given that yesterday I had to actively convince ChatGPT to even pretend something was real for a question.

The moment you tell it “pretend that X is Y” it immediately responds with some variation of “I am an AI trained on real info and can’t imagine things”. If you retry a bunch of times or actually try to convince it (“I understand, but if…”) it eventually complies.

Re: ChatGPT produces made-up nonexistent references

#86

Post by trilbyglens »

wcoenen wrote: post #53
doomlaser wrote: post #30

This is called "hallucination" and I find it to be the biggest flaw with GPT-3 interfaces like ChatGPT. Basically, the system will start convincingly BSing at any point and as a user, you're never sure if any particular detail it outputs is actually correct. The problem is, these large language models are inherently kind of a black box, so how do you fix this kind of behavior? I started looking and it turns out that…

I'm not sure "hallucination" is the right word. I've seen it referred to as "stochastic parroting" elsewhere, and that probably gives more insight into what is happening. These large language models are trained to predict the next word for a given input. And they don't have a choice about this; they must predict the next word, even if it means that they have to make something up. So perhaps the solution would be to i…

Isn't the problem more _because_ it's a language model, and not a knowledge model? It's not based on facts, or even able to go out and find facts. If it's not in the training set it simply doesn't know.

It seems like this is only a single layer to something that should be larger. It should be able to tell if what it's saying is true, or to go out and find facts when it's missing them.

The fact that it's only a language model probably means that this is just out of scope.

Re: ChatGPT produces made-up nonexistent references

#87

Post by tkgally »

In one of the first tests I did with ChatGPT yesterday, I played the role of a person asking for travel advice. Here is one of the exchanges (italics added):

Me: London and New York sound good, because people speak English there. I’ve heard those cities are really expensive, though. Do you know how much it would cost to stay in a cheap but safe hotel in those cities?

ChatGPT: I'm sorry, but I don't have access to current information about hotel prices in London and New York, as my training only goes up until 2021 and I do not have the ability to browse the internet.

This might help to explain all of those phony references.

When I’ve asked for essays that required historical knowledge but no citations, the results were quite accurate. All of my ChatGPT test results (so far) are here:

https://www.gally.net/temp/202212chatgpt/index.html

Re: ChatGPT produces made-up nonexistent references

#88

Post by scotty79 »

doomlaser wrote: post #30

This is called "hallucination" and I find it to be the biggest flaw with GPT-3 interfaces like ChatGPT. Basically, the system will start convincingly BSing at any point and as a user, you're never sure if any particular detail it outputs is actually correct. The problem is, these large language models are inherently kind of a black box, so how do you fix this kind of behavior? I started looking and it turns out that…

I am currently having a lot of fun trying to figure out how some stuff in Rust works by asking GPT.

It gives nice, informative answers, however most of them contain small error. When I point it out it happily agrees that I'm correct and helpfully explains why in fact what it told me before was wrong in that detail.

This actually might be cool way to learn programming. When you have a compiler at hand to check things that seem suspicious to you. It kinda keeps you on your toes.

Re: ChatGPT produces made-up nonexistent references

#89

Post by dsr_ »

ShamelessC wrote: post #57
wcoenen wrote: post #53

Earlier quoted context omitted.

I'm not sure "hallucination" is the right word. I've seen it referred to as "stochastic parroting" elsewhere, and that probably gives more insight into what is happening. These large language models are trained to predict the next word for a given input. And they don't have a choice about this; they must predict the next word, even if it means that they have to make something up. So perhaps the solution would be to i…

Hallucination is commonly used in ML parlance and gets the point across without needing to know what "stochastic" means.

"Stochastic" means "random, not supported by facts, hypothetical" in every context in which it is used, across many fields.

The real problem is that anyone thought that they could pull factual material out of a giant language correlation network.

Re: ChatGPT produces made-up nonexistent references

#90

Post by [deleted] »

djmips wrote: post #59
doomlaser wrote: post #30

This is called "hallucination" and I find it to be the biggest flaw with GPT-3 interfaces like ChatGPT. Basically, the system will start convincingly BSing at any point and as a user, you're never sure if any particular detail it outputs is actually correct. The problem is, these large language models are inherently kind of a black box, so how do you fix this kind of behavior? I started looking and it turns out that…

Fake it till you make it.

[deleted]
Post reply on HN