Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

arrowtsx.dev

Post reply on HN
Print view View original

161–170 of 318 posts

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#161

Post by harrall »

wolttam wrote: post #113

> it is clear that actual intelligence has plateaued significantly. > Moving forward, the industry cannot continue to train bigger and bigger models since their intelligence not only plateaus but often will get worse These are wild claims - why are we concluding that bigger models and more data = more hallucination? That’s actually the opposite of what’s been happening over the last couple years. Some models may stil…

In cognitive science, it appears your brain has two modes of thinking:

- A very parallel type of computation that is fast and generally accurate and integrates hundreds of variables. It’s sometimes labeled as intuition or system 1 thinking.

- A much slower, step by step, analytical type, commonly linked with your pre-frontal cortex (one of the newest parts of the brain). Sometimes called system 2 thinking.

Maybe the way the universe works is that all computation more or less is one of those two types. In which case, an LLM alone is only the first part, which is often right but its results also cannot ever be proven.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#163

Post by grayhatter »

ordersofmag wrote: post #115
grayhatter wrote: post #12

Earlier quoted context omitted.

> In other words, you shouldn't choose the model that hallucinates the least without detailed prompting You're prompting it wrong is quickly becoming the new, you're holding it wrong. It's wild how willing software engineers are to blame the user when the actual problem is their own defective design. Ideally we all, as an industry, will stop accepting this as reasonable excuse for the demonstrated incompetence

It's not that you're prompting it wrong. It's that you're judging the output against a standard (human intelligence) that just isn't relevant--no matter how much we want it to be and no matter how much the fluency of the output tricks us into thinking there's a human-like mind behind it. Now granted, if the boat salesmen were pushing hard on the idea that the boat would fly and even put little wings on the side and I…

> It's not that you're prompting it wrong. It's that you're judging the output against a standard (human intelligence) that just isn't relevant

It's not that you're holding it wrong, you're just wrong for expecting it to work the way it's described (able to one shot most problems these days).

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#164

Post by npilk »

andai wrote: post #136

> GPT-5.5 and DeepSeek V4 Pro are two of the clearest hallucination leaders, despite being absolutely huge. Because of their immense size they simply did not learn how to say “I don’t know” or recognize intricate logical and technical fallacies. This implies that bigger models are more likely to hallucinate? That doesn't match my experience.

I think it implies they are more likely to hallucinate if they don't know the answer. So a big model will return the correct answer more often than a small one, but in the cases where it doesn't, it will be more likely to make something up instead of saying "I don't know".

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#165

Post by techpression »

embedding-shape wrote: post #117
ExtremisAndy wrote: post #104

Earlier quoted context omitted.

See, this, to me, seems obvious, but I’m sure it’s more challenging/complex than I can imagine (I am NOT an expert on AI in any way imaginable). But there has to be a solution. Just yesterday I was asking Gemini to tell me about a certain college professor, and it gave me a list of facts about them. And it was perfect. Then, out of curiosity, I followed up with “tell me more about him!” and it spit out several more b…

I bet most of these issues are essentially system prompt/harness issues. If your example had "Validate any details before sharing them with the user, with multiple sources" as the system prompt, it was using a model that is strong at following system prompts precisely and had access to some basic tools, then it'd spend maybe minutes more, but the answer would have been way more accurate. But no, Google want "the new…

Errors multiply though, you might just get more plausible sounding errors than actual facts.

Like when agent 1 says X, agent 2 verifies it as Y and the original question ends up being some weird amalgamation of Z with additional ”this is really true” statements sprinkled on top.

I agree Google responses hurt more than help, but I’ve also gotten identical outcomes of 40min self-reasoning Opus threads (it’s less common obviously).

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#166

Post by grayhatter »

jpalomaki wrote: post #61
grayhatter wrote: post #8

Earlier quoted context omitted.

> Hallucination rate scores are a little tricky to interpret because they're conditional on the model not knowing the answer. That means they don't measure the probability of your encountering a hallucination in everyday use, since that also depends on the probability of the model not knowing the answer, as well as how well your distribution of tasks aligns with the distribution tested in the eval. Do you have a cite…

As human I also give wrong answers if if I know the right one. Sometimes I also give answers even when I don’t really know them. When pushed, I then start thinking and realise my mistake. System 1 vs 2?

That's weird, why do you do that?

When someone asks a question, if I don't know the answer; I say I don't know.

System 1 vs 2 doesn't really matter... I won't use an LLM that's willing to make up random shit. Equally I also won't work with a human who does that. Trust and confidence a system will function correctly is an important quality, in both humans and genai

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#167

Post by blovescoffee »

mcphage wrote: post #160
jmalicki wrote: post #133

Earlier quoted context omitted.

I find these internet arguments talking about LLMs as if they are trained by reading the internet to be wild. Yes, pretraining still exists. But for the past few years, pretraining by reading the internet is just the initial bootstrapping of LLM training. The RL training they get from bespoke training data, with very very different characteristics than what these armchair analyses claim, dominates these days.

Where do they get the bespoke training data from? And how much? I don’t really know anything about this.

Companies like Mercor sell data from human experts

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#168

Post by czk »

if you're benchmaxxing then maybe bigger doesnt always mean better, but for general intelligence and big model smell, that couldn't be further from the truth

the oss models are impressive but it's pretty clear how quickly they fall off when you try to use them outside of a narrow set of problems they benchmarked well on when compared to opus/5.5

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#169

Post by stevemk14ebr »

harrall wrote: post #161
wolttam wrote: post #113

> it is clear that actual intelligence has plateaued significantly. > Moving forward, the industry cannot continue to train bigger and bigger models since their intelligence not only plateaus but often will get worse These are wild claims - why are we concluding that bigger models and more data = more hallucination? That’s actually the opposite of what’s been happening over the last couple years. Some models may stil…

In cognitive science, it appears your brain has two modes of thinking: - A very parallel type of computation that is fast and generally accurate and integrates hundreds of variables. It’s sometimes labeled as intuition or system 1 thinking. - A much slower, step by step, analytical type, commonly linked with your pre-frontal cortex (one of the newest parts of the brain). Sometimes called system 2 thinking. Maybe the…

An LLM is not thinking, assuming and relating it to thought and universal truths is nonsense.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#170

Post by embedding-shape »

techpression wrote: post #165
embedding-shape wrote: post #117

Earlier quoted context omitted.

I bet most of these issues are essentially system prompt/harness issues. If your example had "Validate any details before sharing them with the user, with multiple sources" as the system prompt, it was using a model that is strong at following system prompts precisely and had access to some basic tools, then it'd spend maybe minutes more, but the answer would have been way more accurate. But no, Google want "the new…

Errors multiply though, you might just get more plausible sounding errors than actual facts. Like when agent 1 says X, agent 2 verifies it as Y and the original question ends up being some weird amalgamation of Z with additional ”this is really true” statements sprinkled on top. I agree Google responses hurt more than help, but I’ve also gotten identical outcomes of 40min self-reasoning Opus threads (it’s less common…

> Like when agent 1 says X, agent 2 verifies it as Y and the original question ends up being some weird amalgamation of Z with additional ”this is really true” statements sprinkled on top.

Yeah, seems what grounds agents right now is quite literally human thoughts and text, so if you're doing something like that, you really need to pass the original user prompt through the entire way, for every "child" to keep in mind the final thing, otherwise it does seem to spiral out of control.

Post reply on HN