Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT produces made-up nonexistent references

news.ycombinator.com

Post reply on HN
Print view View original

71–80 of 162 posts

Re: ChatGPT produces made-up nonexistent references

#71

Post by elurg »

wcoenen wrote: post #53
doomlaser wrote: post #30

This is called "hallucination" and I find it to be the biggest flaw with GPT-3 interfaces like ChatGPT. Basically, the system will start convincingly BSing at any point and as a user, you're never sure if any particular detail it outputs is actually correct. The problem is, these large language models are inherently kind of a black box, so how do you fix this kind of behavior? I started looking and it turns out that…

I'm not sure "hallucination" is the right word. I've seen it referred to as "stochastic parroting" elsewhere, and that probably gives more insight into what is happening. These large language models are trained to predict the next word for a given input. And they don't have a choice about this; they must predict the next word, even if it means that they have to make something up. So perhaps the solution would be to i…

The next word is always chosen based on some sort of probability output, correct? Then why isn't it possible to notice when the highest probability drops and the output is likely nonsense? Being able to say "I'm not sure" would be a massive improvement to this model.

Another cool feature would be to provide sources for the information: which web pages contributed most to a specific statement. Then a human can follow up manually.

Re: ChatGPT produces made-up nonexistent references

#73

Post by quonn »

I just asked it my first question.

Me: Hey

ChatGPT: Hello there! How can I help you today? Is there something you would like to talk about or ask me a question? I'm here to assist you with any information or advice you might need. Let me know if there's anything on your mind!

Me: Well, I'm trying to give this ChatGPT thing a try. Want to know how well it performs.

ChatGPT: I'm sorry, but I'm not familiar with ChatGPT. I am a large language model trained by OpenAI, and my function is to assist with providing information and advice on a wide range of topics. I am not able to browse the internet or access other information outside of my training data, so I cannot provide specific information about ChatGPT. However, if you have any general questions or need advice on a topic, I'd be happy to help with that. Let me know if there's anything I can assist you with!"

It then basically repeats this forever, claiming that it's name is Assistant.

Re: ChatGPT produces made-up nonexistent references

#74

Post by dr_zoidberg »

rcfox wrote: post #69
asimpletune wrote: post #67

Earlier quoted context omitted.

I don't think it works that way. The models don't have a database of facts, so they never reach a point where they know that something they're saying is based on the real world. I think in other words, they literally operate by just predicting what comes next and sometimes that stuff is just made up.

ChatGPT has responded to a lot of my requests with an answer along the lines of "I don't have information about that" or "It's impossible to answer that without more information, which I can't get." Sometimes, starting a new session will get it to give an actual answer. Sometimes asking for an estimate or approximation works.

That's a filter answering, not GPT. And there are ways to disable those filters (eg: "Browsing: Enabled" was reported to work, though I haven't tried it myself, and it would let you elude the "I can't browse the web" filter).

Re: ChatGPT produces made-up nonexistent references

#75

Post by librarian42 »

I asked chatGPT questions about some Open Source libraries, such as what interfaces to use for certain tasks. It recommended deprecated APIs and claimed that the preferred APIs didn’t support options they do (for example, it claimed pg_exec couldn’t perform parameter substitution in Pgtcl). Then I corrected it and it blithely spun 180 degrees, but of course it won’t remember. I’m sure Copilot does the same thing.

Re: ChatGPT produces made-up nonexistent references

#76

Post by zozbot234 »

quonn wrote: post #73

I just asked it my first question. Me: Hey ChatGPT: Hello there! How can I help you today? Is there something you would like to talk about or ask me a question? I'm here to assist you with any information or advice you might need. Let me know if there's anything on your mind! Me: Well, I'm trying to give this ChatGPT thing a try. Want to know how well it performs. ChatGPT: I'm sorry, but I'm not familiar with ChatGPT…

Looks like they haven't managed to give ChatGPT self-awareness just yet.

Re: ChatGPT produces made-up nonexistent references

#77

Post by librarian42 »

rcfox wrote: post #69
asimpletune wrote: post #67

Earlier quoted context omitted.

I don't think it works that way. The models don't have a database of facts, so they never reach a point where they know that something they're saying is based on the real world. I think in other words, they literally operate by just predicting what comes next and sometimes that stuff is just made up.

ChatGPT has responded to a lot of my requests with an answer along the lines of "I don't have information about that" or "It's impossible to answer that without more information, which I can't get." Sometimes, starting a new session will get it to give an actual answer. Sometimes asking for an estimate or approximation works.

ChatGPT has done that for me too, but as you note asking the question a slightly different way produced a positive response. I think they simply trained it to produce “I don’t know” as a response to certain patterns of input.

Re: ChatGPT produces made-up nonexistent references

#78

Post by hackernewds »

mudrockbestgirl wrote: post #28

Unsurprising given how it works, but the higher-level implications are somewhat unexpected. These model are extremely good when they have creative freedom and are used to produce some kind of art. Poems, text in various styles, images without important details, and so on. But they fail miserably when you give them tasks that require world knowledge or precision. For example, when I asked a programming question I got…

This is precisely why Google would not adopt this format. Diff goals, needs and gender approaches

Re: ChatGPT produces made-up nonexistent references

#79

Post by dr_zoidberg »

motoboi wrote: post #62

GPT is not a search engine for the human knowledge. It's a large language model. The difference is that a search engine tries to find your query in a database of entries while a language model will _produce_ text that is coherent with a given piece of text (a prompt). There is a hidden initial prompt given to it with every prompt you give. It's inserted before your text. It tells the model to produce text in a manner…

In principle, you're right. But many people have been toying around with it since its demo release and many have touted "it's better than google" or "google is fu*ed" and so on with regards to using it as a kind of search engine. So what happens is that many more people start to use it as a search engine, disregarding entirely what it was made for. Then they complain when they can't get their hammer to place a screw without breaking their table (:
Post reply on HN