Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

arrowtsx.dev

Post reply on HN
Print view View original

61–70 of 318 posts

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#61

Post by jpalomaki »

grayhatter wrote: post #8
aesthesia wrote: post #4

Hallucination rate scores are a little tricky to interpret because they're conditional on the model not knowing the answer. That means they don't measure the probability of your encountering a hallucination in everyday use, since that also depends on the probability of the model not knowing the answer, as well as how well your distribution of tasks aligns with the distribution tested in the eval. I'd also hesitate to…

> Hallucination rate scores are a little tricky to interpret because they're conditional on the model not knowing the answer. That means they don't measure the probability of your encountering a hallucination in everyday use, since that also depends on the probability of the model not knowing the answer, as well as how well your distribution of tasks aligns with the distribution tested in the eval. Do you have a cite…

As human I also give wrong answers if if I know the right one. Sometimes I also give answers even when I don’t really know them.

When pushed, I then start thinking and realise my mistake. System 1 vs 2?

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#62

Post by embedding-shape »

andybak wrote: post #55
gymbeaux wrote: post #17

Earlier quoted context omitted.

Those numbers are abysmal. Should we really be using LLMs to write our code? I have a theory- LLMs can spit out code that gets the job done and looks ok, maybe even great, but contains small “anomalies” that compound over time. An enterprise app developed entirely with LLM-happy devs might end up virtually unmaintainable. I’m not sure how to explain it, but the more I see LLM-written code the more I feel it’s bad cod…

I can't help but feel that people continually underestimate how bad human written code becomes over time. The exception is probably single-person passion projects or open source projects that maintain quality governance over time. I strongly suspect most closed source code developed under commercial or internal pressure is pretty awful after a few years of development. All LLM code has to do is suck less than existin…

Sucky human-written code is still based on human understanding, which can change over time, be readjusted or solidified. People implement something wrong once, then update their perspective, then in the future does it right.

LLMs doesn't have this benefit. You forget to add the correct to the system prompt, and the LLM will repeat the same mistake over and over, and worse than that, their mistakes aren't based on their understanding, it's basically random guesses.

Humans, even bad coders, still seem to have some sort of architecture in mind, even if it's spaghetti, whereas LLMs (obviously) don't think more than a few steps, and never about the full scope of what they're contributing too, and on purpose too, because you want the context to be as small as possible when you work with LLMs.

With LLMs you need to thread carefully between "What does the LLM need to know?" and "Can I skip passing this to the LLM this time?" while a human you can more or less dump them everything you sit on, and let them shift it through, and they'll mostly make it out OK.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#63

Post by fuck_google »

oshrimpton wrote: post #54
wiether wrote: post #52

Purely anecdotal, but when OpenAI removed Codex-5.3 from the ChatGPT sub and forced me to move to GPT-5.5, the result was far worse than what I was enjoying with Codex. And, of course, it was burning 10 times more tokens for this output.

Yeah they are 100% in the wrong for removing the fine tuned codex models. It makes sense why they wouldn't want to allocate so many resources towards fine tuning but still the enshittification of GPT models is real

[dead]

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#64

Post by embedding-shape »

oshrimpton wrote: post #54
wiether wrote: post #52

Purely anecdotal, but when OpenAI removed Codex-5.3 from the ChatGPT sub and forced me to move to GPT-5.5, the result was far worse than what I was enjoying with Codex. And, of course, it was burning 10 times more tokens for this output.

Yeah they are 100% in the wrong for removing the fine tuned codex models. It makes sense why they wouldn't want to allocate so many resources towards fine tuning but still the enshittification of GPT models is real

Huh, the fine-tuned "codex" variants always seemed like "quick specific edit" prototypes that weren't meant for real use. They worked OK when you were very specific, but besides that, nowhere close to GPT5.X and the other "real" models.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#65

Post by cyanydeez »

stalfie wrote: post #60

One thing I wonder about hallucinations, is that it seems on the surface that it is an easy problem for RLVR to target. Since you're already generating enormous amounts of reasoning traces which are verified by correct answers, just have "don't know" as an option as a valid answer, and on problems where none of the thousands of reasoning traces led to a correct answer, just promote the traces that led to the "don't k…

the problem is the null answer will stop the "markov" chain.

so, thats all.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#66

Post by amelius »

stalfie wrote: post #60

One thing I wonder about hallucinations, is that it seems on the surface that it is an easy problem for RLVR to target. Since you're already generating enormous amounts of reasoning traces which are verified by correct answers, just have "don't know" as an option as a valid answer, and on problems where none of the thousands of reasoning traces led to a correct answer, just promote the traces that led to the "don't k…

But if an LLM says "I don't know" should you pay for the tokens?

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#67

Post by wiether »

embedding-shape wrote: post #64
oshrimpton wrote: post #54

Earlier quoted context omitted.

Yeah they are 100% in the wrong for removing the fine tuned codex models. It makes sense why they wouldn't want to allocate so many resources towards fine tuning but still the enshittification of GPT models is real

Huh, the fine-tuned "codex" variants always seemed like "quick specific edit" prototypes that weren't meant for real use. They worked OK when you were very specific, but besides that, nowhere close to GPT5.X and the other "real" models.

Since Codex-5.3 came out it was my daily driver for everything: quick scripting, greenfield projects, new features on old projects...

Idk if it was the harness (OpenCode), my AGENT or my prompts, but I was getting exactly what I wanted, and quickly.

With GPT-5.5 it tries to play smart, takes much more times and is often stuck on basic stuff that DeepSeek solves oneshot.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#68

Post by xzenor »

andybak wrote: post #55
gymbeaux wrote: post #17

Earlier quoted context omitted.

Those numbers are abysmal. Should we really be using LLMs to write our code? I have a theory- LLMs can spit out code that gets the job done and looks ok, maybe even great, but contains small “anomalies” that compound over time. An enterprise app developed entirely with LLM-happy devs might end up virtually unmaintainable. I’m not sure how to explain it, but the more I see LLM-written code the more I feel it’s bad cod…

I can't help but feel that people continually underestimate how bad human written code becomes over time. The exception is probably single-person passion projects or open source projects that maintain quality governance over time. I strongly suspect most closed source code developed under commercial or internal pressure is pretty awful after a few years of development. All LLM code has to do is suck less than existin…

And where do you think the LLM learned coding from?

But anyway, let the LLM verify the code to give advice on improvements but don't let it write code unverified. That's my opinion on it anyway.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#69

Post by embedding-shape »

wiether wrote: post #67
embedding-shape wrote: post #64

Earlier quoted context omitted.

Huh, the fine-tuned "codex" variants always seemed like "quick specific edit" prototypes that weren't meant for real use. They worked OK when you were very specific, but besides that, nowhere close to GPT5.X and the other "real" models.

Since Codex-5.3 came out it was my daily driver for everything: quick scripting, greenfield projects, new features on old projects... Idk if it was the harness (OpenCode), my AGENT or my prompts, but I was getting exactly what I wanted, and quickly. With GPT-5.5 it tries to play smart, takes much more times and is often stuck on basic stuff that DeepSeek solves oneshot.

> With GPT-5.5 it tries to play smart, takes much more times and is often stuck on basic stuff that DeepSeek solves oneshot.

You have any session logs or similar that shows this thing? Never once, since I started using the codex TUI when it became available, has GPT models gotten stuck on something another model breeze through, I quite literally run every prompt I do through multiple providers, this would be very visible very quickly for me.

I remember trying every -codex variant of the models and could never get them to be productive for tasks taking longer than 5-10 minutes, compared to GPT 5.5 which quite literally worked through the night day (with the /goal feature), and actually had something valuable and useful in the end this morning that wasn't exploding in LOC and complexity. I don't think any of the -codex variants would have been able to do this at all, based on how they worked when I last used them.

Post reply on HN