Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

arrowtsx.dev

Post reply on HN
Print view View original

211–220 of 318 posts

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#211

Post by big_paps »

jpalomaki wrote: post #61
grayhatter wrote: post #8

Earlier quoted context omitted.

> Hallucination rate scores are a little tricky to interpret because they're conditional on the model not knowing the answer. That means they don't measure the probability of your encountering a hallucination in everyday use, since that also depends on the probability of the model not knowing the answer, as well as how well your distribution of tasks aligns with the distribution tested in the eval. Do you have a cite…

As human I also give wrong answers if if I know the right one. Sometimes I also give answers even when I don’t really know them. When pushed, I then start thinking and realise my mistake. System 1 vs 2?

I realized that people from india often show this kind of behaviour in my experience . They superconfidently give you a wrong answer and walk away or even help by making things worse and then dissappear shrugging .. Are you from india ?

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#212

Post by [deleted] »

an0malous wrote: post #121
wolttam wrote: post #113

> it is clear that actual intelligence has plateaued significantly. > Moving forward, the industry cannot continue to train bigger and bigger models since their intelligence not only plateaus but often will get worse These are wild claims - why are we concluding that bigger models and more data = more hallucination? That’s actually the opposite of what’s been happening over the last couple years. Some models may stil…

> why are we concluding that bigger models and more data = more hallucination? That’s not what your quotes said. They said bigger models = plateau in intelligence, nothing about more data or increased hallucinations The relevant quote for what you’re talking about would be: > It’s been proven that when a model is trained on large volumes of highly factual and non-theoretical data, it learns to always have an answer.…

[deleted]

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#213

Post by themgt »

an0malous wrote: post #200
themgt wrote: post #178

Earlier quoted context omitted.

That’s not what your quotes said. They said bigger models = plateau in intelligence, nothing about more data or increased hallucinations ... I’m pretty sure #1 is well known Well known in a multiverse branch where Fable was a dud?

No, well known in the current multiverse branch where we still occasionally use things like math and scientific analysis instead of people’s vibe checks and pelican SVGs. Here’s the paper from OpenAI where Dario himself was a co-author: https://arxiv.org/pdf/2001.08361 > We have observed consistent scalings of language model log-likelihood loss with non-embedding parameter count N, dataset size D, and optimized train…

instead of people’s vibe checks and pelican SVGs.

Right, what happened is everyone went to Fable and asked it to make the very best bicycle pelican SVG, no mistakes. And Fable's bicycle pelican SVGs were such timeless masterpieces, we all instantly got AI psychosis. Happily, you were immune to this.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#214

Post by jmalicki »

mcphage wrote: post #160
jmalicki wrote: post #133

Earlier quoted context omitted.

I find these internet arguments talking about LLMs as if they are trained by reading the internet to be wild. Yes, pretraining still exists. But for the past few years, pretraining by reading the internet is just the initial bootstrapping of LLM training. The RL training they get from bespoke training data, with very very different characteristics than what these armchair analyses claim, dominates these days.

Where do they get the bespoke training data from? And how much? I don’t really know anything about this.

> And how much?

Mercor, one of the larger vendors for contracting with experts to create bespoke data, says on their webpage they're paying $3M/day to their contractors for data.

So well into the billions of dollars a year for bespoke training data.

That's also ignoring the RLVR data labs can get from software - they can use the vibe coding sessions as training data as well without paying more.

They are just one of many.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#215

Post by andybak »

layer8 wrote: post #209
andybak wrote: post #55

Earlier quoted context omitted.

I can't help but feel that people continually underestimate how bad human written code becomes over time. The exception is probably single-person passion projects or open source projects that maintain quality governance over time. I strongly suspect most closed source code developed under commercial or internal pressure is pretty awful after a few years of development. All LLM code has to do is suck less than existin…

That doesn’t help the developers who have high standards.

Yes. But that's not the point I'm addressing.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#216

Post by macleginn »

jmalicki wrote: post #210
MattRogish wrote: post #206

Earlier quoted context omitted.

I'm not saying they are not trying - I'm saying we're inventing new problems faster than any Lab can: 1) Identify the gaps 2) Determine how to fix them 3) Implement a fix (especially if that fix is: identify and find experts) 4) And judge the result How do they know [person] is an expert in [some field]? How do they find that person? How many experts are necessary to give the right information? How do we evaluate the…

> How do they know [person] is an expert in [some field]? How do they find that person? They have a PhD from a top school, they are a licensed attorney, they are a licensed physician, a board certified cardiologist, etc. They are constantly recruiting from these populations with well-paying side gigs. > 4) And judge the result That's what they pay the experts for. And to have experts review the other experts with pee…

> Noone is really caring about hallucinations on point facts these days though, it is much more about complex reasoning tasks.

The boundary is pretty thin there though. E.g., Gemini recently told me that a certain papers claims that two frameworks are mathematically equivalent, while the paper shows the opposite, and yesterday Google's AI overview told me that no World Cup matches were scheduled for that day despite their being several of them. The model probably used complex reasoning to arrive at both (incorrect) answers, but superficially they look like basic errors of fact.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#217

Post by jmalicki »

macleginn wrote: post #216
jmalicki wrote: post #210

Earlier quoted context omitted.

> How do they know [person] is an expert in [some field]? How do they find that person? They have a PhD from a top school, they are a licensed attorney, they are a licensed physician, a board certified cardiologist, etc. They are constantly recruiting from these populations with well-paying side gigs. > 4) And judge the result That's what they pay the experts for. And to have experts review the other experts with pee…

> Noone is really caring about hallucinations on point facts these days though, it is much more about complex reasoning tasks. The boundary is pretty thin there though. E.g., Gemini recently told me that a certain papers claims that two frameworks are mathematically equivalent, while the paper shows the opposite, and yesterday Google's AI overview told me that no World Cup matches were scheduled for that day despite…

That is a great example of the kind of thing they're paying people to create as training data.

You write the prompt, and then write rubrics to judge the responses, and you found something the model failed at. Congratulations, you just earned $500, now do it again.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#218

Post by jgalt212 »

jmalicki wrote: post #133
an0malous wrote: post #121

Earlier quoted context omitted.

> why are we concluding that bigger models and more data = more hallucination? That’s not what your quotes said. They said bigger models = plateau in intelligence, nothing about more data or increased hallucinations The relevant quote for what you’re talking about would be: > It’s been proven that when a model is trained on large volumes of highly factual and non-theoretical data, it learns to always have an answer.…

I find these internet arguments talking about LLMs as if they are trained by reading the internet to be wild. Yes, pretraining still exists. But for the past few years, pretraining by reading the internet is just the initial bootstrapping of LLM training. The RL training they get from bespoke training data, with very very different characteristics than what these armchair analyses claim, dominates these days.

Outside of games and coding generating enough valid examples and counter-examples to harness the power of RL is cost prohibitive.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#219

Post by solid_fuel »

SeriousM wrote: post #94
solid_fuel wrote: post #3

> It’s been proven that when a model is trained on large volumes of highly factual and non-theoretical data, it learns to always have an answer. DeepSeek V4 Pro (1.6T params, 49B active, 44 AA Intelligence Index score) has a ludicrous 94% hallucination score on the AA-Omniscience benchmark, meaning on questions that it couldn’t figure out, it only stated that it didn’t know around 6% of the time, and the rest it conf…

There is no concept of "knowledge" in LLM as it is on Wikipedia. The question-tokens define the answer-tokens. That's it. The art relies in clustering the relevant weights together.

Correct, LLMs are not ontologically capable of “knowing”. That is why I put “know” in quotes.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#220

Post by hereme888 »

Artificial Analysis says GPT-5.5 xhigh scores highest on AA-Omniscience accuracy. The article focuses on rate instead of overall accuracy. Those are different things: a model can answer more questions correctly overall while still being worse at abstaining when wrong.

Curiously, this post and article is the only submission and interaction the OP has made, and these claims support the product he's intending to release.

Post reply on HN