Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

arrowtsx.dev

Post reply on HN
Print view View original

291–300 of 318 posts

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#291

Post by coldtea »

andybak wrote: post #262
8note wrote: post #231

Earlier quoted context omitted.

> while a human you can more or less dump them everything you sit on, and let them shift it through, and they'll mostly make it out OK i dont see why software engineers are paid so well, and are so hard to hire? just dump a bunch of requirements on a homeless person and itll just work out

I have no idea what point you're making here.

He's making the point that the idea that "with a human you can more or less dump them everything you sit on, and let them shift it through, and they'll mostly make it out OK" is wrong.

Humans absolutely need lots of training to be able to get this dump and do it right, and even when trained, different humans or humans at different coding tasks can be totally way worse and way better at it.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#292

Post by macleginn »

jmalicki wrote: post #263
macleginn wrote: post #259

Earlier quoted context omitted.

Yes, they do have money to burn, and this will bring some improvements for sure, but active learning has never really worked out, has it? And even 10% of the educated population doing this for, like, 50 years is not that much data, while normally each accuracy percentage is more and more data-expensive.

Eh, maybe 20% of my ML career earnings has been showing that active learning works out. shrug

Ok, cool, depends on application, probably.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#293

Post by claytongulick »

coldtea wrote: post #286
claytongulick wrote: post #197

Earlier quoted context omitted.

My impression is that the fundamental issue is that LLMs attempt to extract reasoning (executive execution) from data (relationship between tokens). There's an open question about whether this is theoretically possible, but it doesn't seem like it to me. Human generated data is an effect of reasoning. Attempting to extract executive function from it is kind of like taking an anti-derivative of a function. This has al…

> The original intelligence that created those tokens was driven by a whole universe of inputs, from hormones to starlight to gravity Still inputs, that in the end changed something about synapses and their activation. And whether doesn't have a strong enough local effect to be material to the those operations, can be ignored too. E.g. gravity might kill you via a fall or a tide drowning you, but might have zero infl…

> but might have zero influence in your thinking at the brain operation level

Most of the people that I follow that research this stuff at the doctoral level say that they (and modern science) have very little idea about how consciousness works, the mind, and its relationship to the physical parts of the brain.

We know about a lot of relationships, we can damage parts of the brain and see that damage expressed through changed behavior, but there appears to be far more to it than just a wet neural network of synapses and neurotransmitters that can be accurately modeled with a set of floating point numbers.

Also, as an aside, gravity plays a much larger role in how the brain operates than may be immediately obvious [1].

[1] https://pmc.ncbi.nlm.nih.gov/articles/PMC9650717/

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#294

Post by [deleted] »

sometimelurker wrote: post #241
remix2000 wrote: post #148

Earlier quoted context omitted.

Well the difference here is that you're overly simplifying complex biology and many other factors whereas llms are in fact actually simple mathematical models. As always, the devil lies in the details. Dismissing intricacies is a useful tool for daydreamers, not so much for engineers.

LLMs actually aren't simple Markov chains tho, your also simplifying. and LLMs trained with RLVR aren't just optimized over the space of functions (like gpt2 was), they're optimized over the space of programs (programs under some length). You find the ideal algorithm that can do the task you need it to.

[deleted]

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#295

Post by stcg »

In the referenced benchmark GLM-5.2 (max) got 25% of all questions correct. GPT-5.5 (xhigh) got 57% correct.

https://artificialanalysis.ai/evaluations/omniscience

I'd much rather have some answer that I can verify than no answer to verify.

I don't want a model that says "I don't know", because I will verify the answer anyway.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#296

Post by bwfan123 »

stcg wrote: post #295

In the referenced benchmark GLM-5.2 (max) got 25% of all questions correct. GPT-5.5 (xhigh) got 57% correct. https://artificialanalysis.ai/evaluations/omniscience I'd much rather have some answer that I can verify than no answer to verify. I don't want a model that says "I don't know", because I will verify the answer anyway .

> I don't want a model that says "I don't know", because I will verify the answer anyway.

Few people actually review answers or code. Because they have been sold the myth that these models can do it all. The main problem is that LLMs dont have causal models, and as a result, their reasoning is a high probability word salad and not a logically sound argument. Particularly on tricky corner cases which it hasnt encountered. I would still agree with you that sometimes hallucinations are actually useful as it provides a strawman, and having even a hallucinated answer to spar with is better than a "dont know".

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#297

Post by HarHarVeryFunny »

jmalicki wrote: post #251
jgalt212 wrote: post #247

Earlier quoted context omitted.

still cost prohibitive.

Yes, which is why for some things I've gotten paid as much as $1500 per training example generated. AI labs don't care about cost prohibitive.

Could you give a specific example - just the outline/shape of what it consisted of (or more detail if you care to)?

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#298

Post by coldtea »

claytongulick wrote: post #293
coldtea wrote: post #286

Earlier quoted context omitted.

> The original intelligence that created those tokens was driven by a whole universe of inputs, from hormones to starlight to gravity Still inputs, that in the end changed something about synapses and their activation. And whether doesn't have a strong enough local effect to be material to the those operations, can be ignored too. E.g. gravity might kill you via a fall or a tide drowning you, but might have zero infl…

> but might have zero influence in your thinking at the brain operation level Most of the people that I follow that research this stuff at the doctoral level say that they (and modern science) have very little idea about how consciousness works, the mind, and its relationship to the physical parts of the brain. We know about a lot of relationships, we can damage parts of the brain and see that damage expressed throug…

The gravity in the paper, as far as I can see, is about the influence of microgravity (e.g. much much smaller than regular Earth gravity) influence in blood circulation, bodily fluids etc, and as such on sleep, breathing, heart function, and other such aspects.

Not about whether gravity affects the synapses and neurotransmitters in their regular function.

We know of course that e.g. bad sleep or lack of water affects the brain too, as do many other factors. The effects of microgravity studied there are of similar nature.

My argument wasn't that such a lack of gravity, or such other factors, can't affect it in that sense, but that it doesn't play a role on the actual mechanics of synapses and neurotransmitters (in other works, they could be modelled without it).

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#299

Post by gymbeaux »

wizzledonker wrote: post #159
andybak wrote: post #55

Earlier quoted context omitted.

I can't help but feel that people continually underestimate how bad human written code becomes over time. The exception is probably single-person passion projects or open source projects that maintain quality governance over time. I strongly suspect most closed source code developed under commercial or internal pressure is pretty awful after a few years of development. All LLM code has to do is suck less than existin…

I think the real issue might be that how “good” the code is matters less than being able to form a mental model for what the human who wrote the code was “thinking”. If written by a machine, this contract is broken and we get more confused, even if our traditional methods of evaluating the code come out equal.

Yes, thank you for wording it better. When I read through for example an entire codebase that was ~99% written by AI, it's "inconsistent" in a way that even a shared-by-humans codebase would not be. I think this arises from the AI misunderstanding slightly what is being asked - the AI misunderstands, but can still (at least in some cases) output code that does what it needs to. It may also do other things that it doesn't need to do, or may do the thing in a suboptimal, not-so-maintainable way, but the UI works and that's enough for most non-technical people.

Re: GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2

#300

Post by gymbeaux »

xzenor wrote: post #68
andybak wrote: post #55

Earlier quoted context omitted.

I can't help but feel that people continually underestimate how bad human written code becomes over time. The exception is probably single-person passion projects or open source projects that maintain quality governance over time. I strongly suspect most closed source code developed under commercial or internal pressure is pretty awful after a few years of development. All LLM code has to do is suck less than existin…

And where do you think the LLM learned coding from? But anyway, let the LLM verify the code to give advice on improvements but don't let it write code unverified. That's my opinion on it anyway.

If I have to verify the code then I don't see a point in using it to write more than a single method at a time, and that method should be simple enough that I can take a very quick glance and be able to tell it's correct - something like a method that writes an array to a CSV. I don't have that code memorized, especially in the various languages I regularly work in, but I know it when I see it. Anything more complex than that and I think it would take me as much or more time to truly verify the AI's output than to just write it myself.
Post reply on HN