Hey, here is a frog that can talk! Ask it anything! - Human: Ok frog, how much is 1 + 1? - Frog: The answer to your question is 3. - Human: Meh, wrong answer. Not impressed ..
[deleted]
ChatGPT produces made-up nonexistent references
101–110 of 162 posts
Re: ChatGPT produces made-up nonexistent references
#102This is called "hallucination" and I find it to be the biggest flaw with GPT-3 interfaces like ChatGPT. Basically, the system will start convincingly BSing at any point and as a user, you're never sure if any particular detail it outputs is actually correct. The problem is, these large language models are inherently kind of a black box, so how do you fix this kind of behavior? I started looking and it turns out that…
I am currently having a lot of fun trying to figure out how some stuff in Rust works by asking GPT. It gives nice, informative answers, however most of them contain small error. When I point it out it happily agrees that I'm correct and helpfully explains why in fact what it told me before was wrong in that detail. This actually might be cool way to learn programming. When you have a compiler at hand to check things…
Indeed. I asked it to implement quicksort in x86 assembly, but increment each element by 5. It made a mistake: instead of incrementing by 5, it replaced it with a 5 (mov insted of add). I pointed it out, it agreed with me, elaborated on what is the proper solution and provided it. Its weird.
Re: ChatGPT produces made-up nonexistent references
#103This is called "hallucination" and I find it to be the biggest flaw with GPT-3 interfaces like ChatGPT. Basically, the system will start convincingly BSing at any point and as a user, you're never sure if any particular detail it outputs is actually correct. The problem is, these large language models are inherently kind of a black box, so how do you fix this kind of behavior? I started looking and it turns out that…
I'm not sure "hallucination" is the right word. I've seen it referred to as "stochastic parroting" elsewhere, and that probably gives more insight into what is happening. These large language models are trained to predict the next word for a given input. And they don't have a choice about this; they must predict the next word, even if it means that they have to make something up. So perhaps the solution would be to i…
> I'm not sure "hallucination" is the right word. I've seen it referred to as "stochastic parroting" elsewhere, and that probably gives more insight into what is happening.
It may give more insight, but it seems to me that hallucination is very similar: the brain completing some incomplete/random data to what it thinks is plausible and/or desirable.
Re: ChatGPT produces made-up nonexistent references
#104GTP has always been the worlds best bullshit simulator. Its output is incredible until you look closely and realize that it’s not so much inaccurate, but completely hallucinated. It’s the kind of thing that would be great for filling in Lorem Ipsum placeholder text that’s just convincing enough if no one looks too close.
> just convincing enough That’s basically the principle of the Turing test. But maybe the GPT approach is fundamentally incapable of withstanding very detailed scrutiny.
Just ask it what day it is. No detailed scrutiny needed.
Re: ChatGPT produces made-up nonexistent references
#105Earlier quoted context omitted.
I am currently having a lot of fun trying to figure out how some stuff in Rust works by asking GPT. It gives nice, informative answers, however most of them contain small error. When I point it out it happily agrees that I'm correct and helpfully explains why in fact what it told me before was wrong in that detail. This actually might be cool way to learn programming. When you have a compiler at hand to check things…
Indeed. I asked it to implement quicksort in x86 assembly, but increment each element by 5. It made a mistake: instead of incrementing by 5, it replaced it with a 5 (mov insted of add). I pointed it out, it agreed with me, elaborated on what is the proper solution and provided it. Its weird.
Maybe it needs some sort of automatic internal plausibility check — reflecting on its own output. Pointing out its mistake basically triggers that, for the particular aspect that is being pointed out.
Humans probably do something similar. Sometimes you notice that what you’re uttering doesn’t completely make sense, or is missing some bit to be comprehensible. Then you correct yourself or elaborate.
Re: ChatGPT produces made-up nonexistent references
#106Unsurprising given how it works, but the higher-level implications are somewhat unexpected. These model are extremely good when they have creative freedom and are used to produce some kind of art. Poems, text in various styles, images without important details, and so on. But they fail miserably when you give them tasks that require world knowledge or precision. For example, when I asked a programming question I got…
I was playing around with ChatGPT yesterday and what I found was: - Ask a straightforward question like what are business models for open source software or what are the advantages of GPL vs. AGPL vs. MIT and the answers were pretty shallow and boilerplate but they were correct as far as they went. - But ask a question like "Why didn't large SMP systems become more mainstream?" (which basically requires synthesizing…
Also, if you’d ask it “Why did large SMP systems become mainstream?", it would (presumably) spit out a plausible-sounding explanation for that. The models seem to take the prompt as a premise, and then go from there. Maybe that’s also why when you point out a mistake, they oblige and provide a correction.
I haven’t tried: If you point out something they said as a mistake when in fact it was correct, do they notice and object? Maybe if the evidence from the training data is strong enough.
Re: ChatGPT produces made-up nonexistent references
#107Unsurprising given how it works, but the higher-level implications are somewhat unexpected. These model are extremely good when they have creative freedom and are used to produce some kind of art. Poems, text in various styles, images without important details, and so on. But they fail miserably when you give them tasks that require world knowledge or precision. For example, when I asked a programming question I got…
The art isn't that good either, it's just less obviously wrong(plus there's subjective taste involved).
The bad code is good enough to fool a code illiterate person and the art is good enough to fool artistically ignorant people( which is most of us).
Still huge progress though!
Re: ChatGPT produces made-up nonexistent references
#108If you ask it the lyrics to 'Uptown Girl', it'll completely make some lines up.
Re: ChatGPT produces made-up nonexistent references
#109I've seen Copilot do the same thing - if you leave a comment like "// see this StackOverflow post", it'll often complete the comment with a link that looks a lot like a SO post but is actually nonsense. My guess is that GPT-3 has seen a lot of SO links (or other links) and understands that sometimes engineers find it handy to have a long url-looking-thing with some numbers at the end, but it doesn't understand the fa…
Yeah, I once saw Copilot add a comment that talked about a workaround for bug in the win32 API it was calling, referencing a stackoverflow post.
Obviously, the stackoverflow post didn't exit. I googled, and the bug it described didn't exist. Even widening the search, I couldn't find any documentation about any bug in that API. The emitted code after the comment was pretty good and didn't contain a workaround.
It just hallucinated the comment about a workaround because it's seen other code do similar things and thought it would look nice here.
I find Copilot to be a useful tool... But it is not a knowledge engine and should not be trusted. In my opinion, you only accept it's suggestions if you were about to type the same thing.
Re: ChatGPT produces made-up nonexistent references
#110Earlier quoted context omitted.
I was playing around with ChatGPT yesterday and what I found was: - Ask a straightforward question like what are business models for open source software or what are the advantages of GPL vs. AGPL vs. MIT and the answers were pretty shallow and boilerplate but they were correct as far as they went. - But ask a question like "Why didn't large SMP systems become more mainstream?" (which basically requires synthesizing…
Also, if you’d ask it “Why did large SMP systems become mainstream?", it would (presumably) spit out a plausible-sounding explanation for that. The models seem to take the prompt as a premise, and then go from there. Maybe that’s also why when you point out a mistake, they oblige and provide a correction. I haven’t tried: If you point out something they said as a mistake when in fact it was correct, do they notice an…
Interesting. I'll have to try that. And, yes, you are correct. It gave a short pablum answer that pretty much focused on the benefits of multiple cores.
The mistake in the original was somewhat subtle and gets into programming for large SMP vs. MPP/distributed systems. And it's actually true that parallel programming even on shared memory systems has challenges (and there was a lot of discussion at one point when multicore was coming in on desktops) but it's pretty much flat wrong to say that we didn't have large SMP systems primarily because they were hard to program for.