Live data from Hacker News

ChatGPT produces made-up nonexistent references

news.ycombinator.com

111–120 of 162 posts

Re: ChatGPT produces made-up nonexistent references

#111
post #103
post #53

Earlier quoted context omitted.

I'm not sure "hallucination" is the right word. I've seen it referred to as "stochastic parroting" elsewhere, and that probably gives more insight into what is happening. These large language models are trained to predict the next word for a given input. And they don't have a choice about this; they must predict the next word, even if it means that they have to make something up. So perhaps the solution would be to i…

> I'm not sure "hallucination" is the right word. I've seen it referred to as "stochastic parroting" elsewhere, and that probably gives more insight into what is happening. It may give more insight, but it seems to me that hallucination is very similar: the brain completing some incomplete/random data to what it thinks is plausible and/or desirable.

That's how sensory processing work in general, not just hallucinations.

Re: ChatGPT produces made-up nonexistent references

#112
I hate to spell out the obvious but this isn't a bug, it's an incomplete feature.

GPT5 will generate the rest of the text and upload it to the arxiv, then submit it to the relevant peer reviewed publications.

The reviewers will also be GPT5 trained for accuracy. They will suggest a few edits to avoid obvious lies, plus will suggest the author cite the reviewer's friends to preserve mimesis. Accept with revision.

The papers will come out, h-indices will increase, the authors will get tenure, and all will be well. This niche of humanity can retire to a well deserved rest and relaxation -- kind of a backdoor to universal basic income.

Re: ChatGPT produces made-up nonexistent references

#113
post #104
post #29

Earlier quoted context omitted.

> just convincing enough That’s basically the principle of the Turing test. But maybe the GPT approach is fundamentally incapable of withstanding very detailed scrutiny.

Just ask it what day it is. No detailed scrutiny needed.

I wonder if GPT models can get super precise or if there’s a fundamental limitation.

Re: ChatGPT produces made-up nonexistent references

#115

Earlier quoted context omitted.

In principle, you're right. But many people have been toying around with it since its demo release and many have touted "it's better than google" or "google is fu*ed" and so on with regards to using it as a kind of search engine. So what happens is that many more people start to use it as a search engine, disregarding entirely what it was made for. Then they complain when they can't get their hammer to place a screw…

This has been driving me crazy on twitter lately and it really just goes to show how little scrutiny many people give to work that "looks correct" and matches whatever goal / output they were expecting. On so many of the "college kids will never write another essay" posts if you dig into the comments there is usually at least one subject matter expert who read the essay pointing out a bunch of errors and poor reasoni…

Guess what, actual school essays also tend to have a bunch of errors and poor reasoning... especially those by kids who would be inclined to cheat rather than write their own essay.

Sure, you can nitpick for errors, but do you really think something like this wouldn't cut it at school? We are not talking about writing technical papers or laws, but about writing at a relatively novice level...

> Write a school essay on why you should never order pizza with pineapple.

> Pizza is a beloved food enjoyed by people all over the world, but one topping that has been the subject of much debate is pineapple. While some people argue that pineapple adds a sweet and tropical flavor to the pizza, others believe that it is a misguided and unacceptable topping that ruins the inherent goodness of the pizza. In this essay, I will argue that pineapple should never be ordered on a pizza, and explain why it is an abomination in the eyes of true pizza lovers.

First and foremost, pineapple does not belong on pizza because it goes against the fundamental principles of pizza making. Pizza is a delicate balance of flavors, with the savory and slightly tangy sauce, the rich and salty cheese, and the crisp and chewy crust all coming together to create a perfect symphony of flavor. Adding pineapple to the mix disrupts this balance, and the sweetness of the fruit clashes with the other flavors in an unpleasant and unappetizing way.

Furthermore, pineapple does not have the same texture as the other toppings commonly found on pizza. The traditional pizza toppings, such as pepperoni, mushrooms, and olives, have a firm and slightly chewy texture that complements the crust and the sauce. Pineapple, on the other hand, is soft and mushy, and it can make the pizza soggy and unappealing. The different textures of the pineapple and the other toppings can also be off-putting, and can make the pizza difficult to eat.

In addition to the flavor and texture issues, pineapple on pizza is simply a gross and unappetizing concept. The combination of pineapple and pizza is unnatural and bizarre, and it goes against everything that pizza stands for. Pizza is a savory and satisfying food, meant to be enjoyed with friends and family, and adding pineapple to it just makes it weird and unappealing.

In conclusion, pineapple should never be ordered on a pizza. It goes against the fundamental principles of pizza making, it has a texture that does not complement the other toppings, and it is simply gross and unappetizing.

Re: ChatGPT produces made-up nonexistent references

#116

Hey, here is a frog that can talk! Ask it anything! - Human: Ok frog, how much is 1 + 1? - Frog: The answer to your question is 3. - Human: Meh, wrong answer. Not impressed ..

You seem to imply that the reaction is unreasonable, but if one is invited to ask anything it seems a fair expectation that the reply should be an actual answer, rather than plausible-sounding noises superficially associated with the context.

As for the fantastic frog, getting words out of it at all would be a remarkable achievement for the different reason of how unnatural it would be for a frog to produce those particular sounds, but neither storing binary sequences that can be programmatically mapped to symbols nor retrieving or performing arithmetic on such sequences is exceptional use of a computer program.

Re: ChatGPT produces made-up nonexistent references

#117

Earlier quoted context omitted.

This has been driving me crazy on twitter lately and it really just goes to show how little scrutiny many people give to work that "looks correct" and matches whatever goal / output they were expecting. On so many of the "college kids will never write another essay" posts if you dig into the comments there is usually at least one subject matter expert who read the essay pointing out a bunch of errors and poor reasoni…

Guess what, actual school essays also tend to have a bunch of errors and poor reasoning... especially those by kids who would be inclined to cheat rather than write their own essay. Sure, you can nitpick for errors, but do you really think something like this wouldn't cut it at school? We are not talking about writing technical papers or laws, but about writing at a relatively novice level... > Write a school essay o…

At the high school level especially?

I just asked it to write an essay on the Battle of Agincourt. Perfectly serviceable prose that looks accurate to me. I'm sure I could flesh it out using some follow-on questions. What were the causes? Describe the reign of Henry V. Describe the English tactics and why they won. Etc.

Add a few convincing references to any of the doubtless many books on the subject for sale on Amazon and I could have a perfectly adequate high school or maybe even low-level college essay in an hour tops.

(And I did run some random sentences through Google looking for exact matches and didn't find them.)

The Battle of Agincourt was a significant event in English and French history, as it was a major victory for the English army against the French during the Hundred Years' War. The battle took place on October 25, 1415, near the town of Agincourt in northern France.

The English army, led by King Henry V, was vastly outnumbered by the French forces, which were estimated to be around 30,000 soldiers compared to the English army of just 6,000. Despite the odds, the English were able to emerge victorious thanks to their superior tactics and weaponry.

The English army was composed of mostly longbowmen, who were able to shoot arrows with great accuracy and power. The French, on the other hand, relied heavily on heavily-armored knights, who were less effective in the muddy and wet conditions of the battlefield. The English were also able to take advantage of the narrow front of the battlefield, which allowed them to focus their firepower and overwhelm the French forces.

The battle lasted for several hours and was marked by intense hand-to-hand combat. In the end, the English emerged victorious, with the French suffering heavy casualties. The English lost only a few hundred soldiers, while the French lost thousands.

The victory at Agincourt was a major turning point in the Hundred Years' War, as it weakened the French army and boosted English morale. It also cemented King Henry V's reputation as a great military leader and helped to solidify English control over parts of France.

The Battle of Agincourt is still remembered today as a key event in English and French history, and it continues to be celebrated in popular culture, such as in Shakespeare's play "Henry V." Overall, it was a significant military victory for the English and a crucial moment in the history of the Hundred Years' War.

Re: ChatGPT produces made-up nonexistent references

#118
Well I personally know one of those people whose name GPT took the liberty of attributing to a made-up work. It's a mild example, but really puts into perspective the concerns about relying on this garbage for any important thing. What happens when I ask it "give me a list of pedophiles"?

Or less dramatically: what happens when it attributes mistaken results or offensive quotes to a real person, damaging his/her reputation?

Re: ChatGPT produces made-up nonexistent references

#119
Same for music recommendations. I've told ChatGPT some of my taste and asked for some song recommendations.

It replied with some songs from actual artists that I like, perfectly describing how the song's tone was, why I'd like it, how the vocals fitted the tone, and the release date of the song.

Great... the only problem was those songs didn't exist.

Re: ChatGPT produces made-up nonexistent references

#120
post #53

Earlier quoted context omitted.

I'm not sure "hallucination" is the right word. I've seen it referred to as "stochastic parroting" elsewhere, and that probably gives more insight into what is happening. These large language models are trained to predict the next word for a given input. And they don't have a choice about this; they must predict the next word, even if it means that they have to make something up. So perhaps the solution would be to i…

Isn't the problem more _because_ it's a language model, and not a knowledge model? It's not based on facts, or even able to go out and find facts. If it's not in the training set it simply doesn't know. It seems like this is only a single layer to something that should be larger. It should be able to tell if what it's saying is true, or to go out and find facts when it's missing them. The fact that it's only a langua…

That doesn't stop the companies churning out these models to pretend otherwise x)
Post reply on HN