GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2 • Hacker News Board

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#231

Post by 8note » Sat, Jun 20, 2026, 6:34 PM UTC

embedding-shape wrote: post #62

andybak wrote: post #55

Earlier quoted context omitted.

I can't help but feel that people continually underestimate how bad human written code becomes over time. The exception is probably single-person passion projects or open source projects that maintain quality governance over time. I strongly suspect most closed source code developed under commercial or internal pressure is pretty awful after a few years of development. All LLM code has to do is suck less than existin…

Sucky human-written code is still based on human understanding, which can change over time, be readjusted or solidified. People implement something wrong once, then update their perspective, then in the future does it right. LLMs doesn't have this benefit. You forget to add the correct to the system prompt, and the LLM will repeat the same mistake over and over, and worse than that, their mistakes aren't based on the…

> while a human you can more or less dump them everything you sit on, and let them shift it through, and they'll mostly make it out OK

i dont see why software engineers are paid so well, and are so hard to hire?

just dump a bunch of requirements on a homeless person and itll just work out

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#232

Post by x3cca » Sat, Jun 20, 2026, 6:40 PM UTC

giancarlostoro wrote: post #177

I wonder if this is what a “Minimally Viable LLM” looks like. I often wonder how much of an LLM do you need before you can just shove a bigger context Window and any dynamic knowledge content to it like a PDF or markdown file to give it knowledge outside of its training data. I feel like LLMs don’t need more data they just need to be refined.

You might be interested in this model. It's a densely trained on math whuch let's it punch way higher than it should https://github.com/WeiboAI/VibeThinker

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#233

Post by grayhatter » Sat, Jun 20, 2026, 6:41 PM UTC

aesthesia wrote: post #15

grayhatter wrote: post #8

Earlier quoted context omitted.

> Hallucination rate scores are a little tricky to interpret because they're conditional on the model not knowing the answer. That means they don't measure the probability of your encountering a hallucination in everyday use, since that also depends on the probability of the model not knowing the answer, as well as how well your distribution of tasks aligns with the distribution tested in the eval. Do you have a cite…

This isn't quite the point. When comparing two different models' hallucination rates, the denominator is different. The evaluation works more or less like this: for each question, the model has the option to answer or abstain, so there are three possible outcomes: the model answers and gets it right, the model answers and gets it wrong (hallucination), or the model abstains. The hallucination rate is (model answers w…

The way you define the evaluation criteria seems very problematic[^1].

I don't understand the point of describing it as 3 possible outcomes. I objected to it because the only reason I would do something like that would be to obscure the severity of the model defects. I'm sure I'm missing something, but the reason I suspect that's how it's done, is to [intentionally] obscure the actual meaningful metrics.

I would expect any engineer to evaluate any model using accuracy, (error rate), and usefulness (definitive answer rate), as strictly independent metrics. Did it answer, and if it answered, did emit incorrect or misleading information and how many quantifiable bits of each.

The false negative rate (model confirmed to contain the requested output/information via other method but was unable to for the given test) is significant, but given a non-definitive answer is significantly different from a definitive and incorrect answer. Why would you want to group hallucinations?

Number/rate of useful answers (correct and incorrect) and error rate (given any answer how often will that answer be defective in some way).

To be clear, I'm differentiating hallucination rate from eagerness to answer, even though they're obviously linked because I believe presenting 20 correct answers, 20 incorrect answers, and 60 abstentions as a hallucination rate of 25% as obviously malicious. If I give you 40 answers, 20 correct and 20 incorrect. the error rate is 50% and if it refused an additional 60 times, it's usefulness rate would be 40%... arguably 20% depending on how strict you choose to be about the definition of useful. The matrix we should be using is a 2x2 true positive, false positive, true negative, false negative. But being that honest that might make the model look bad!

[^1]: just in case it's unclear, I'm using you exclusively rhetorically. I don't think you personally are being misleading, only that you're explaining how it's done... but that's the problem isn't it.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#234

Post by gcanyon » Sat, Jun 20, 2026, 6:43 PM UTC

> it is clear that actual intelligence has plateaued significantly

N=1, but I disagree strongly. I'm writing a hard-science science fiction story, and the physics of it is at (and frankly, beyond) my skillset. The story's plot has had to change over a dozen times as I realized errors in my application of physics in the story.

Throughout, I've been reviewing the physics with LLMs, mainly Gemini 3.1 Pro Preview, but also with Claude and OpenAI. Often I have the LLMs debate each other -- "My friend [another model] said XYZ about the physics, is that right or wrong?" In almost all cases, Gemini explains why the other models are wrong, and when I send its explanation to them, they concede it is right and they are wrong.

As I said, I did the above checks literally dozens of times as I wrote the story. And everything was dialed in: no further issues claimed by anyone, me or the LLMs.

Not with Fable. I managed to get it to review the story while it was running, and it listed out something like ten issues: some minor, some general knowledge-based, and two that were impressive:

1. It pointed out where Gemini (and I, and other LLMs) had missed a , resulting in values about 152 times larger than they should have been. I sent that to Gemini and it fully conceded that it had been wrong all along. 2. It pointed out a simple inconsistency in the application of special relativity (I thought I had that at least dialed in, but no :-/ ) that affected a very specific plot point. The story is novella-length, about 28,000 words long, and this is a point that was mentioned in the first two pages, and then not again until the very last page. And it's obvious, once you realize it. And I missed it. Gemini missed it. Claude and ChatGPT missed it.

Only Fable found it. Again, N=1, but that was a remarkable run I got out of it in the couple days it was available.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#235

Post by nathan_compton » Sat, Jun 20, 2026, 6:48 PM UTC

Synthesizing a bunch of stuff I've read here lately, it seems like if OpenAI and Claude have actually found product market fit (generating code) then the question of hallucination is going to get less attention in the future. If the real money is in code generation (where there is a relatively clear acceptance criteria of at least "it runs and does what I wanted as far as I can tell") then there doesn't seem to be a lot of juice in pulling ones hair out on hallucination of facts.

It seems like for agentic coding, just making sure the AI can find the relevant documentation to establish a ground truth is probably sufficient.

Note that I'm distinguishing here between hallucination of what you might call "free facts" and hallucination of material which deviates from what is in the context itself. The latter seems both a tractable problem and one which will improve coding agent functionality. But the former seems like its no longer on the critical path, probably because its hard.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#236

Post by nathan_compton » Sat, Jun 20, 2026, 6:49 PM UTC

bilater wrote: post #227

wolttam wrote: post #113

> it is clear that actual intelligence has plateaued significantly. > Moving forward, the industry cannot continue to train bigger and bigger models since their intelligence not only plateaus but often will get worse These are wild claims - why are we concluding that bigger models and more data = more hallucination? That’s actually the opposite of what’s been happening over the last couple years. Some models may stil…

Yeah not only is it totally unsubstantiated, the benchmarks are getting less useful to really show the difference between these models. Big model smell is still a thing and GLM 5.2 while impressive is not Fable class. Here is something I would like people to chew on. Perhaps the smartest researchers in the world across multiple labs know more about this than we do? Perhaps they are aware of issues like the data wall…

Are the smartest researchers in the world out there saying there isn't a wall? I don't know of any people doing the actual R&D who frequently make outrageous claims.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#237

Post by bilater » Sat, Jun 20, 2026, 7:07 PM UTC

nathan_compton wrote: post #236

bilater wrote: post #227

Earlier quoted context omitted.

Yeah not only is it totally unsubstantiated, the benchmarks are getting less useful to really show the difference between these models. Big model smell is still a thing and GLM 5.2 while impressive is not Fable class. Here is something I would like people to chew on. Perhaps the smartest researchers in the world across multiple labs know more about this than we do? Perhaps they are aware of issues like the data wall…

Are the smartest researchers in the world out there saying there isn't a wall? I don't know of any people doing the actual R&D who frequently make outrageous claims.

https://x.com/polynoamial/status/2064210146558136827

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#238

Post by djvdq » Sat, Jun 20, 2026, 7:25 PM UTC

bilater wrote: post #237

nathan_compton wrote: post #236

Earlier quoted context omitted.

Are the smartest researchers in the world out there saying there isn't a wall? I don't know of any people doing the actual R&D who frequently make outrageous claims.

https://x.com/polynoamial/status/2064210146558136827

I'd say that as OpenAI employee he's kinda biased on the topic

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#239

Post by sometimelurker » Sat, Jun 20, 2026, 7:31 PM UTC

metalspot wrote: post #77

hallucination is good for tasks that have an external oracle like computer programming

ehh ur right but there's a lot of nuance here. if you have a system that doesn't hallucinate a ton and is still very "creative" that's great, and probably much better than a hallucinating system regardless of its creativity. I'm reminded of theoremproving LLMs working in lean producing millions of slop proofs until one works, but if you have something like that simple RLVR should fix it (external oracle can be the judge for the RL.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#240

Post by ecommerceguy » Sat, Jun 20, 2026, 7:35 PM UTC

It's very much looking like OpenAI will be bailed out, along with all the other Capex'ers. I say this because the trump admin (I feel partially at fault because I voted for him) has indicated they will be bailing out the entire ai stargate from intel and amd to amazon and anthropic. I know alot of everyday folks that absolutely hate - passionately HATE - anything and everything tech bro. Downvote all you want, that's the reality. They see Palantir et al as evil and demonic.