Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

arrowtsx.dev

Post reply on HN
Print view View original

81–90 of 318 posts

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#81

Post by dezgeg »

ben_w wrote: post #48
VBprogrammer wrote: post #46

Earlier quoted context omitted.

> Can there be bugs? Sure. That's the price of not reading or understanding every line. I've yet to come across a human developer who's output would meet this standard, despite writing every line. In fact, having an LLM review our code is catching quite a few bugs before it reaches QA.

Indeed, though I find the distribution is different. The humans may skip unit tests and need reminding; the AI always write unit tests once it's in AGENTS.md or whatever, but my experience* was that 5-10% of the time the LLM's attempt at a "test" would, instead of executing the code and examining the results, open the source code as a text file and run a regex to find/exclude certain substrings . * At the start of th…

I have seen some pre-AI over-mocked codebases where the "tests" where essentially that (but harder to read than regex would have been)

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#82

Post by BDPW »

mmmattt wrote: post #75
guerrilla wrote: post #71

Earlier quoted context omitted.

Why not? It did the work. Why should you expect it to be omniscient? We can rank them based on how much they know and people will gravitate towards those that do know more. It's a market after all.

If it’s a market, wouldn’t the incentive be to lie about knowing and thus to keep the hallucinations?

If you had an llm that could accurately predict when a claim is uncertain it would be very popular, I think. I would pay for that kind of reliability tbh

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#83

Post by macleginn »

stalfie wrote: post #60

One thing I wonder about hallucinations, is that it seems on the surface that it is an easy problem for RLVR to target. Since you're already generating enormous amounts of reasoning traces which are verified by correct answers, just have "don't know" as an option as a valid answer, and on problems where none of the thousands of reasoning traces led to a correct answer, just promote the traces that led to the "don't k…

The main problem here is that hallucination suppression doesn’t generalise. We can penalise models for incorrect answers on a wide range of questions, but this doesn’t lead to the emergence of a coherent worldview, which, coupled with logical abilities, is the only true remedy against hallucinations. With current architectures, hallucinations will likely persist on open-domain tasks forever.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#84

Post by Greenpants »

mmmattt wrote: post #75
guerrilla wrote: post #71

Earlier quoted context omitted.

Why not? It did the work. Why should you expect it to be omniscient? We can rank them based on how much they know and people will gravitate towards those that do know more. It's a market after all.

If it’s a market, wouldn’t the incentive be to lie about knowing and thus to keep the hallucinations?

Up to the point where consumers notice and decide to stop using these models because of it.

Might be why we're already rarely seeing models output an "I don't know".

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#85

Post by CuriouslyC »

aubanel wrote: post #74

> Bigger is not better The article uses the example of GLM being smaller than DeepSeek, yet better on hallucinations as "smaller can be good too" But the GLM family itself is scaling up fast: GLM-5.x family is 754B, double the previous generation of GLM-4.x > comes within just 4 points of GPT-5.5 and 9 points of Fable 5 9 percentage points IS a big difference

If we're hand waiving how an open source model from a Chinese lab that you can use a nearly unlimited amount for <100/mo's 9% difference from the premier, unavailable, expensive when it was available American frontier model, we've already lost.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#86

Post by flexagoon »

spwa4 wrote: post #37

Why is everyone expecting LLMs to be like the Star Trek computer? I wonder if anyone's ever measured what the hallucination rate of a human is.

Because AI company executives and devoted vibecoders constantly make egregious claims like "programming is fully solved" and even straight up "hallucinations don't exist on frontier models"

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#88

Post by embedding-shape »

[deleted] wrote: post #87

[deleted]

> This is a terrible line of thought

They're basing this all on public benchmarks which stopped being a reliable indicator of anything the last 2-3 years. Of course it'll be filled with more terrible lines of thoughts.

People really need to stop placing such importance on public benchmarks. They're valuable for comparing very close models, useful to evaluate if quantization and similar have negative impact, but you're not gonna be able to tell if one model is better than the other based on one scoring a few percentage points higher than a completely different model.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#89

Post by joka88xj »

gymbeaux wrote: post #17
aesthesia wrote: post #4

Hallucination rate scores are a little tricky to interpret because they're conditional on the model not knowing the answer. That means they don't measure the probability of your encountering a hallucination in everyday use, since that also depends on the probability of the model not knowing the answer, as well as how well your distribution of tasks aligns with the distribution tested in the eval. I'd also hesitate to…

Those numbers are abysmal. Should we really be using LLMs to write our code? I have a theory- LLMs can spit out code that gets the job done and looks ok, maybe even great, but contains small “anomalies” that compound over time. An enterprise app developed entirely with LLM-happy devs might end up virtually unmaintainable. I’m not sure how to explain it, but the more I see LLM-written code the more I feel it’s bad cod…

[flagged]

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#90

Post by intended »

BDPW wrote: post #82
mmmattt wrote: post #75

Earlier quoted context omitted.

If it’s a market, wouldn’t the incentive be to lie about knowing and thus to keep the hallucinations?

If you had an llm that could accurately predict when a claim is uncertain it would be very popular, I think. I would pay for that kind of reliability tbh

This would break reality. There’s some underlying physical law that prevents the existence of any algorithm of truth.
Post reply on HN