GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

Post by omneity » Sat, Jun 20, 2026, 9:42 PM UTC

jonathanhefner wrote: post #154

omneity wrote: post #96

Earlier quoted context omitted.

It’s not as simple. I trained an LLM before on exactly this, to scratch the itch of this question. The task was simple, using the MS-MARCO[0] dataset which contains queries, search results, answers, I made a training set that has: 1. Questions paired with real results supporting them (mixed with some irrelevant results), and a correct answer 2. Questions paired only with irrelevant results, with the answer “No answer…

Thank you for sharing! Based on your experience, do you think a two-model system might fare better? For example, two models in serial where the second model is trained to "sniff out" potential hallucinations and fact check them (and possibly iterate with the first model)?

Post by jmalicki » Sat, Jun 20, 2026, 10:16 PM UTC

macleginn wrote: post #259

jmalicki wrote: post #254

Earlier quoted context omitted.

Right now, this is a 10-figure run rate industry. They are generating a lot of this. Also remember it's not just quantity, it's roughly active learning - they're paying for training data that's at the classification boundary, which is way more valuable. I have gotten offers for contracts for full time jobs at high rates with AI labs to do this. Meta has reallocated a lot of their full time SWE staff to do this. All o…

Yes, they do have money to burn, and this will bring some improvements for sure, but active learning has never really worked out, has it? And even 10% of the educated population doing this for, like, 50 years is not that much data, while normally each accuracy percentage is more and more data-expensive.

Post by YeGoblynQueenne » Sat, Jun 20, 2026, 11:14 PM UTC

MattRogish wrote: post #181

jmalicki wrote: post #133

Earlier quoted context omitted.

I find these internet arguments talking about LLMs as if they are trained by reading the internet to be wild. Yes, pretraining still exists. But for the past few years, pretraining by reading the internet is just the initial bootstrapping of LLM training. The RL training they get from bespoke training data, with very very different characteristics than what these armchair analyses claim, dominates these days.

I'd have to imagine there are wildly diminishing marginal returns to additional SFT/post-training passes. There are a bounded number of (useful) derivations/combinations of Duff's device. If Frontier Labs wish to reduce hallucinations on factual things, they will have to hire people (or the data providers will need to) to do fundamental research above and beyond what is available in extant literature and the web. IE…

Post by jazzdev » Sat, Jun 20, 2026, 11:35 PM UTC

jmalicki wrote: post #133

an0malous wrote: post #121

Earlier quoted context omitted.

> why are we concluding that bigger models and more data = more hallucination? That’s not what your quotes said. They said bigger models = plateau in intelligence, nothing about more data or increased hallucinations The relevant quote for what you’re talking about would be: > It’s been proven that when a model is trained on large volumes of highly factual and non-theoretical data, it learns to always have an answer.…

I find these internet arguments talking about LLMs as if they are trained by reading the internet to be wild. Yes, pretraining still exists. But for the past few years, pretraining by reading the internet is just the initial bootstrapping of LLM training. The RL training they get from bespoke training data, with very very different characteristics than what these armchair analyses claim, dominates these days.

Post by qwery2 » Sun, Jun 21, 2026, 1:06 AM UTC

sometimelurker wrote: post #241

remix2000 wrote: post #148

Earlier quoted context omitted.

Well the difference here is that you're overly simplifying complex biology and many other factors whereas llms are in fact actually simple mathematical models. As always, the devil lies in the details. Dismissing intricacies is a useful tool for daydreamers, not so much for engineers.

LLMs actually aren't simple Markov chains tho, your also simplifying. and LLMs trained with RLVR aren't just optimized over the space of functions (like gpt2 was), they're optimized over the space of programs (programs under some length). You find the ideal algorithm that can do the task you need it to.