Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

arrowtsx.dev

Post reply on HN
Print view View original

71–80 of 318 posts

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#71

Post by guerrilla »

amelius wrote: post #66
stalfie wrote: post #60

One thing I wonder about hallucinations, is that it seems on the surface that it is an easy problem for RLVR to target. Since you're already generating enormous amounts of reasoning traces which are verified by correct answers, just have "don't know" as an option as a valid answer, and on problems where none of the thousands of reasoning traces led to a correct answer, just promote the traces that led to the "don't k…

But if an LLM says "I don't know" should you pay for the tokens?

Why not? It did the work. Why should you expect it to be omniscient?

We can rank them based on how much they know and people will gravitate towards those that do know more.

It's a market after all.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#72

Post by rienbdj »

gymbeaux wrote: post #17
aesthesia wrote: post #4

Hallucination rate scores are a little tricky to interpret because they're conditional on the model not knowing the answer. That means they don't measure the probability of your encountering a hallucination in everyday use, since that also depends on the probability of the model not knowing the answer, as well as how well your distribution of tasks aligns with the distribution tested in the eval. I'd also hesitate to…

Those numbers are abysmal. Should we really be using LLMs to write our code? I have a theory- LLMs can spit out code that gets the job done and looks ok, maybe even great, but contains small “anomalies” that compound over time. An enterprise app developed entirely with LLM-happy devs might end up virtually unmaintainable. I’m not sure how to explain it, but the more I see LLM-written code the more I feel it’s bad cod…

I have a theory that LLM generated code in a highly modular style (simple data, pure functions) will be easier to “recover” by a human team when the LLM gets muddled. So Haskell, basically.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#73

Post by O5vYtytb »

andybak wrote: post #55
gymbeaux wrote: post #17

Earlier quoted context omitted.

Those numbers are abysmal. Should we really be using LLMs to write our code? I have a theory- LLMs can spit out code that gets the job done and looks ok, maybe even great, but contains small “anomalies” that compound over time. An enterprise app developed entirely with LLM-happy devs might end up virtually unmaintainable. I’m not sure how to explain it, but the more I see LLM-written code the more I feel it’s bad cod…

I can't help but feel that people continually underestimate how bad human written code becomes over time. The exception is probably single-person passion projects or open source projects that maintain quality governance over time. I strongly suspect most closed source code developed under commercial or internal pressure is pretty awful after a few years of development. All LLM code has to do is suck less than existin…

I've been sent code from vendors that didn't even compile, long before llms were a thing. Most shops that aren't primarily software have really really terrible software.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#74

Post by aubanel »

> Bigger is not better

The article uses the example of GLM being smaller than DeepSeek, yet better on hallucinations as "smaller can be good too"

But the GLM family itself is scaling up fast: GLM-5.x family is 754B, double the previous generation of GLM-4.x

> comes within just 4 points of GPT-5.5 and 9 points of Fable 5

9 percentage points IS a big difference

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#75

Post by mmmattt »

guerrilla wrote: post #71
amelius wrote: post #66

Earlier quoted context omitted.

But if an LLM says "I don't know" should you pay for the tokens?

Why not? It did the work. Why should you expect it to be omniscient? We can rank them based on how much they know and people will gravitate towards those that do know more. It's a market after all.

If it’s a market, wouldn’t the incentive be to lie about knowing and thus to keep the hallucinations?

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#76

Post by csomar »

gymbeaux wrote: post #17
aesthesia wrote: post #4

Hallucination rate scores are a little tricky to interpret because they're conditional on the model not knowing the answer. That means they don't measure the probability of your encountering a hallucination in everyday use, since that also depends on the probability of the model not knowing the answer, as well as how well your distribution of tasks aligns with the distribution tested in the eval. I'd also hesitate to…

Those numbers are abysmal. Should we really be using LLMs to write our code? I have a theory- LLMs can spit out code that gets the job done and looks ok, maybe even great, but contains small “anomalies” that compound over time. An enterprise app developed entirely with LLM-happy devs might end up virtually unmaintainable. I’m not sure how to explain it, but the more I see LLM-written code the more I feel it’s bad cod…

Easy fix: Code's basically free now, so just pipe your errors straight into an LLM and get instant patches. Sure, the patches themselves are broken too, but no worries! just pipe those back in again. Code's disposable now, fresh code generated on every request.

On a more serious note, I think the problem will be the inability to handle/maintain the systems once they are too big and nobody has no idea what's inside of them or what they do.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#80

Post by BDPW »

cyanydeez wrote: post #65
stalfie wrote: post #60

One thing I wonder about hallucinations, is that it seems on the surface that it is an easy problem for RLVR to target. Since you're already generating enormous amounts of reasoning traces which are verified by correct answers, just have "don't know" as an option as a valid answer, and on problems where none of the thousands of reasoning traces led to a correct answer, just promote the traces that led to the "don't k…

the problem is the null answer will stop the "markov" chain. so, thats all.

You dont have to literally send a null token. Train it to generate text that summarizes the evidence that is there but the uncertainty of the final answer to a prompt.
Post reply on HN