Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

arrowtsx.dev

Post reply on HN
Print view View original

241–250 of 318 posts

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#241

Post by sometimelurker »

remix2000 wrote: post #148
hit8run wrote: post #111

Earlier quoted context omitted.

You are also just a variant of markov chains wired in your brain. So what you complaining about?

Well the difference here is that you're overly simplifying complex biology and many other factors whereas llms are in fact actually simple mathematical models. As always, the devil lies in the details. Dismissing intricacies is a useful tool for daydreamers, not so much for engineers.

LLMs actually aren't simple Markov chains tho, your also simplifying. and LLMs trained with RLVR aren't just optimized over the space of functions (like gpt2 was), they're optimized over the space of programs (programs under some length). You find the ideal algorithm that can do the task you need it to.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#242

Post by Bolwin »

gcanyon wrote: post #234

> it is clear that actual intelligence has plateaued significantly N=1, but I disagree strongly. I'm writing a hard-science science fiction story, and the physics of it is at (and frankly, beyond) my skillset. The story's plot has had to change over a dozen times as I realized errors in my application of physics in the story. Throughout, I've been reviewing the physics with LLMs, mainly Gemini 3.1 Pro Preview, but al…

Hah, I noticed the same thing writing fiction with fable. Most models seem to go into a sort of "storytelling mode" where they forget their PhD level smarts. I had a character who is doing repair on a satellite. Most models would give you a half-baked explanation with some technical terms - half of them right half of them wrong.

Fable gave a description so deep that even I couldn't figure out what was going on and had to ask it to give me a simpler explanation.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#243

Post by jmalicki »

nomel wrote: post #230
giardini wrote: post #225

Earlier quoted context omitted.

jmalicki says many things, among them being "As a side gig, I write novel software that solves problems no existing software does," and "Yes, they are hiring the experts themselves. To create new knowledge above and beyond what's on the internet. To be locked away as LLM training data." More likely you're joking and/or paranoid!8-))

> I write novel software that solves problems no existing software does This is actually really easy to do if you step out of web/gui/crud and into something where you won't find public code, most ever, because it's trade secret. For example, manufacturing.

There is also an endless fountain of things you come across every day and think "oh, wouldn't this complex solution to this low priority problem be cool", but noone ever implements it because it's too complex and the problem is low priority.

Anyone writing software for long enough has a long list of these things in the back of their head that are great fodder for LLM training data.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#244

Post by maxnevermind »

jmalicki wrote: post #210
MattRogish wrote: post #206

Earlier quoted context omitted.

I'm not saying they are not trying - I'm saying we're inventing new problems faster than any Lab can: 1) Identify the gaps 2) Determine how to fix them 3) Implement a fix (especially if that fix is: identify and find experts) 4) And judge the result How do they know [person] is an expert in [some field]? How do they find that person? How many experts are necessary to give the right information? How do we evaluate the…

> How do they know [person] is an expert in [some field]? How do they find that person? They have a PhD from a top school, they are a licensed attorney, they are a licensed physician, a board certified cardiologist, etc. They are constantly recruiting from these populations with well-paying side gigs. > 4) And judge the result That's what they pay the experts for. And to have experts review the other experts with pee…

That is informative, I was suspecting that is how models improve their performance on some convoluted "non-googlabe" benchmarks like SimpleBench, that is how, they just got the taste of those those questions from publicly available samples and then hired people to generate similar questions and provide answers for them.

I wonder if extracting those static reasoning chains make sense given a Rich Sutton's "The Bitter Lesson" and Geoffrey Hinton's "People should stop training radiologists now.". I guess until participants make money they won't stop, not sure if they do, so far it is more about expectation of profitability as I understand.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#245

Post by msdz »

utkuumur wrote: post #183
coffeefirst wrote: post #176

Earlier quoted context omitted.

Yeah #2 may be incidental. Suppose one lab focused on bigger, and another on reinforcement training geared towards factual accuracy over sycophancy. You could easily wind up with a model from the second lab that is less powerful but more accurate. I can’t prove it but I suspect there’s a bit of that going on.

I think one problem is that the models that hallucinate often, a few times out of 8 or 16 so that they get good results on benchmarks, most of which measures success out of top k. From benchmark perspective, you don't really care whether 15 of yours 16 generations failed, as long as one succeeded, but as a user you mostly care that 1 out of 16 you get is actually the successful one. I think this effects is more easy…

> it hallucinates like crazy but looks like its by design to boost benchmarks.

Wasn’t there a discussion around some new-ish benchmark _punishing_ hallucination answers (over not replying at all) recently? Maybe in the not-so-distant future, this “spam replies until one’s correct” strategy won’t be able to game a benchmark much at all anymore.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#246

Post by aurareturn »

dominotw wrote: post #198
aurareturn wrote: post #172

Earlier quoted context omitted.

Aren't hallucinations also heavily influenced by compute and memory capacity? IE. Companies can spend more time to verify results in an agentic format, spend more thinking tokens, and less quantization. All of these heavily depend on compute and memory but are proven to decrease hallucinations. Maybe GPT 5.5 is heavily nerfed due to lack of compute, memory, and energy? I agree that it's farfetched to conclude that bi…

article specifically talks about this. deepseek spending significant test time with worse results than klm

So GLM is just a better model than DS then?

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#247

Post by jgalt212 »

jmalicki wrote: post #221
jgalt212 wrote: post #218

Earlier quoted context omitted.

Outside of games and coding generating enough valid examples and counter-examples to harness the power of RL is cost prohibitive.

Which is why rubrics as rewards are used.

still cost prohibitive.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#248

Post by [deleted] »

maxnevermind wrote: post #244
jmalicki wrote: post #210

Earlier quoted context omitted.

> How do they know [person] is an expert in [some field]? How do they find that person? They have a PhD from a top school, they are a licensed attorney, they are a licensed physician, a board certified cardiologist, etc. They are constantly recruiting from these populations with well-paying side gigs. > 4) And judge the result That's what they pay the experts for. And to have experts review the other experts with pee…

That is informative, I was suspecting that is how models improve their performance on some convoluted "non-googlabe" benchmarks like SimpleBench, that is how, they just got the taste of those those questions from publicly available samples and then hired people to generate similar questions and provide answers for them. I wonder if extracting those static reasoning chains make sense given a Rich Sutton's "The Bitter…

[deleted]

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#249

Post by jmalicki »

maxnevermind wrote: post #244
jmalicki wrote: post #210

Earlier quoted context omitted.

> How do they know [person] is an expert in [some field]? How do they find that person? They have a PhD from a top school, they are a licensed attorney, they are a licensed physician, a board certified cardiologist, etc. They are constantly recruiting from these populations with well-paying side gigs. > 4) And judge the result That's what they pay the experts for. And to have experts review the other experts with pee…

That is informative, I was suspecting that is how models improve their performance on some convoluted "non-googlabe" benchmarks like SimpleBench, that is how, they just got the taste of those those questions from publicly available samples and then hired people to generate similar questions and provide answers for them. I wonder if extracting those static reasoning chains make sense given a Rich Sutton's "The Bitter…

There is one level that these training data give examples of specific static reasoning chains.

Given exposure to enough reasoning chains, with training data that is designed around adversarial reasoning and teaching models to reason, these types of training data might be key to teaching models to reason beyond what they could gather from static data.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#250

Post by knollimar »

jmalicki wrote: post #243
nomel wrote: post #230

Earlier quoted context omitted.

> I write novel software that solves problems no existing software does This is actually really easy to do if you step out of web/gui/crud and into something where you won't find public code, most ever, because it's trade secret. For example, manufacturing.

There is also an endless fountain of things you come across every day and think "oh, wouldn't this complex solution to this low priority problem be cool", but noone ever implements it because it's too complex and the problem is low priority. Anyone writing software for long enough has a long list of these things in the back of their head that are great fodder for LLM training data.

I make a trello board to direct spare tokens at when I'm bored now!
Post reply on HN