Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT produces made-up nonexistent references

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 162 posts

Re: ChatGPT produces made-up nonexistent references

#61

Post by nonrandomstring »

sigmoid10 wrote: post #27
laserbeam wrote: post #21

Earlier quoted context omitted.

The biggest problem is the internet doesn't usually look too close

Exactly. This just accurately mirrors social media as a whole. Producing and reading fake content has become a cornerstone of humanity.

Perhaps Sturgeon was right that 90% of everything is rubbish, but the speed with which fake content can be generated threatens to make 99.9% of everything rubbish.

There are several scenarios for this technology to escalate.

Back in the day it was fun to auto-respond to someone's mail auto-responder. The winner was whoever had the biggest hard-drive to hold a mail queue.

It is merely a matter of time before companies realise they can replace people in PR, HR, customer service and other "public facing" roles with GPT-like technologies. At the same time the public are going to arm themselves with counter-technologies, as browser and mail client plugins. There's obviously a "market" for job application bots, auction bidding, ticket and holiday booking bots for scarce resources. But these will get sophisticated in their attempts to trick human users employed to filter and select.

Once the "battle of the bots" is underway we'll see a rapid increase in essentially deceptive, nonsense traffic swamping systems. Real human activity may become impossible to discern in that digital environment.

Re: ChatGPT produces made-up nonexistent references

#62

Post by motoboi »

GPT is not a search engine for the human knowledge.

It's a large language model.

The difference is that a search engine tries to find your query in a database of entries while a language model will _produce_ text that is coherent with a given piece of text (a prompt).

There is a hidden initial prompt given to it with every prompt you give. It's inserted before your text. It tells the model to produce text in a manner of an virtual assistant that answer questions.

So it's more like role playing. The produced text will be coherent with a question an answers text. Even with the answers are not "real"

Re: ChatGPT produces made-up nonexistent references

#64

Post by lvncelot »

doomlaser wrote: post #30

This is called "hallucination" and I find it to be the biggest flaw with GPT-3 interfaces like ChatGPT. Basically, the system will start convincingly BSing at any point and as a user, you're never sure if any particular detail it outputs is actually correct. The problem is, these large language models are inherently kind of a black box, so how do you fix this kind of behavior? I started looking and it turns out that…

What I find interesting is that hallucination is a big flaw in models like GPT when viewed as a conversational partner that is grounded in reality, while it's actually an achievement when, e.g., synthesizing art via GAN - where the exact opposite, a mere summation or sampling of the source material is what is being avoided.

Re: ChatGPT produces made-up nonexistent references

#65

Post by gtsnexp »

I got a very different output for the exact same prompt:

"I'm sorry, but I am a large language model trained by OpenAI and do not have access to academic references or other external information. My knowledge is based on the text that I was trained on, which has a fixed cutoff point in time. I do not have the ability to browse the internet or access new information that may have been published since my training data was collected.

However, I can tell you that in mathematics, a list is a sequence of values that are typically written within square brackets and separated by commas. For example, a list of numbers might look like this: [1, 2, 3, 4, 5].

Lists have many mathematical properties, such as the ability to be sorted, concatenated, and indexed. They can also be used in various mathematical operations, such as computing the sum or product of all the elements in the list.

In general, lists are a useful tool in mathematics, as they allow you to organize and manipulate data in a structured way. There are many books and other resources that discuss the mathematical properties of lists in greater detail, so if you are interested in learning more, you may want to consult those sources."

Re: ChatGPT produces made-up nonexistent references

#66

Post by rzzzt »

wcoenen wrote: post #53
doomlaser wrote: post #30

This is called "hallucination" and I find it to be the biggest flaw with GPT-3 interfaces like ChatGPT. Basically, the system will start convincingly BSing at any point and as a user, you're never sure if any particular detail it outputs is actually correct. The problem is, these large language models are inherently kind of a black box, so how do you fix this kind of behavior? I started looking and it turns out that…

I'm not sure "hallucination" is the right word. I've seen it referred to as "stochastic parroting" elsewhere, and that probably gives more insight into what is happening. These large language models are trained to predict the next word for a given input. And they don't have a choice about this; they must predict the next word, even if it means that they have to make something up. So perhaps the solution would be to i…

If you would like another Latin word for it, take "confabulation" from neuroscience-land: https://en.wikipedia.org/wiki/Confabulation#Signs_and_sympto...

Re: ChatGPT produces made-up nonexistent references

#67

Post by asimpletune »

andix wrote: post #60
wcoenen wrote: post #53

Earlier quoted context omitted.

I'm not sure "hallucination" is the right word. I've seen it referred to as "stochastic parroting" elsewhere, and that probably gives more insight into what is happening. These large language models are trained to predict the next word for a given input. And they don't have a choice about this; they must predict the next word, even if it means that they have to make something up. So perhaps the solution would be to i…

They are not forced to come up with new ideas. They can also write something like „I have no further information about that“. But in training this is probably discouraged, because they shouldn’t answer all questions like that.

I don't think it works that way. The models don't have a database of facts, so they never reach a point where they know that something they're saying is based on the real world. I think in other words, they literally operate by just predicting what comes next and sometimes that stuff is just made up.

Re: ChatGPT produces made-up nonexistent references

#69

Post by rcfox »

asimpletune wrote: post #67
andix wrote: post #60

Earlier quoted context omitted.

They are not forced to come up with new ideas. They can also write something like „I have no further information about that“. But in training this is probably discouraged, because they shouldn’t answer all questions like that.

I don't think it works that way. The models don't have a database of facts, so they never reach a point where they know that something they're saying is based on the real world. I think in other words, they literally operate by just predicting what comes next and sometimes that stuff is just made up.

ChatGPT has responded to a lot of my requests with an answer along the lines of "I don't have information about that" or "It's impossible to answer that without more information, which I can't get."

Sometimes, starting a new session will get it to give an actual answer. Sometimes asking for an estimate or approximation works.

Re: ChatGPT produces made-up nonexistent references

#70

Post by nonameiguess »

wcoenen wrote: post #53
doomlaser wrote: post #30

This is called "hallucination" and I find it to be the biggest flaw with GPT-3 interfaces like ChatGPT. Basically, the system will start convincingly BSing at any point and as a user, you're never sure if any particular detail it outputs is actually correct. The problem is, these large language models are inherently kind of a black box, so how do you fix this kind of behavior? I started looking and it turns out that…

I'm not sure "hallucination" is the right word. I've seen it referred to as "stochastic parroting" elsewhere, and that probably gives more insight into what is happening. These large language models are trained to predict the next word for a given input. And they don't have a choice about this; they must predict the next word, even if it means that they have to make something up. So perhaps the solution would be to i…

That still wouldn't help here. We don't want the prediction confidence that the sequence of words you produced might appear in a valid English-language sentence produced by humans. We want the prediction confidence that the sentence is factually accurate. These models aren't given that kind of data to train on and I'm not sure how they even could be. There are oodles and oodles of human-generated text out there, but little in the way of verification regarding how much of it is true, to say nothing of categories of language like imperative and artistic that don't have truth values at all.
Post reply on HN