Making o1, o3, and Sonnet 3.7 hallucinate for everyone • Hacker News Board

Re: Making o1, o3, and Sonnet 3.7 hallucinate for everyone

#121

Post by skerit » Sat, Mar 01, 2025, 10:10 PM UTC

dominicq wrote: post #5

ChatGPT used to assure me that you can use JS dot notation to access elements in a Python dict. It also invented Redocly CLI flags that don't exist. Claude sometimes invents OpenAPI specification rules. Any time I ask anything remotely niche, LLMs are often bad.

> Any time I ask anything remotely niche, LLMs are often bad

As soon as the AI coder tools (like Aider, Cline, Claude-Coder) come into contact with a _real world_ codebase, it does not end well.

So far I think they managed to fix 2 relatively easy issues on their own, but in other cases they: - Rewrote tests in a way that the broken behaviour passes the test - Fail to solve the core issue in the code, and instead patch-up the broken result (Like `if (result.includes(":") || result.includes("?")) { /* super expensive stupid fixed for a single specific case */ }` - Failed to even update the files properly, wasting a bunch of tokens

Re: Making o1, o3, and Sonnet 3.7 hallucinate for everyone

#122

Post by xmprt » Sat, Mar 01, 2025, 10:13 PM UTC

simion314 wrote: post #117

andix wrote: post #36

Earlier quoted context omitted.

I try to keep "boring" code to a minimum, by finding meaningful and simple abstractions. LLMs are especially bad handling those, because they were not trained on non-standard abstractions. Edit: most LLMs are great for spitting out some code that fulfills 90% of what you asked for. That's sometimes all you need. But we all know that the last 10% usually take the same amount of effort as the first 90%.

>most LLMs are great for spitting out some code that fulfills 90% of what you asked for. That's sometimes all you need. But we all know that the last 10% usually take the same amount of effort as the first 90%. The issue is if you have an LLm write for you 10k lines of code, where 100 lines are bugged. Now you need to debug the code you did not write and find the bugged code, you will waste similar amount of time. Th…

In my experience using LLMs, the 90% is less about buggy code and more about just ignoring 10% of the features that you require. So it will write code that's mostly correct in 100-1000 lines of code (not buggy) but then no matter how hard you try, it won't get the remaining 10% right and in the process, it will mess up parts of the 90% that was already working or end up writing another 1000 lines of undecipherable code to get 97% there but still never 100% unless you're building something that's not that unique.

Re: Making o1, o3, and Sonnet 3.7 hallucinate for everyone

#123

Post by jonas21 » Sat, Mar 01, 2025, 10:16 PM UTC

latexr wrote: post #43

> Conclusion > LLMs are really smart most of the time. No, the conclusion is they’re never “smart”. All they do is regurgitate text which resembles a continuation of what came before, and sometimes—but with zero guarantees—that text aligns with reality.

Same as you and me, really.

Re: Making o1, o3, and Sonnet 3.7 hallucinate for everyone

#124

Post by MortyWaves » Sat, Mar 01, 2025, 10:18 PM UTC

QuantumGood wrote: post #65

andix wrote: post #30

I've got a lot of hallucinations like that from LLMs. I really don't get how so many people can get LLMs to code most of their tasks without those issues permanently popping up.

GPT can't even tell what its done or give what it knows it should. It's endless, "Apologies, here is what you actually asked for ..." and again it isn't.

This was my primary reason for using Claude. Absolutely useless experience with chatgpt oftentimes. I’ve mainly been using LLMs to help maintain a ridiculously poorly made technical debt dumpster fire, and Claude has been really helpful here mainly with repetitive code.

Re: Making o1, o3, and Sonnet 3.7 hallucinate for everyone

#125

Post by [deleted] » Sat, Mar 01, 2025, 10:18 PM UTC

hombre_fatal wrote: post #76

Terr_ wrote: post #47

Earlier quoted context omitted.

> I use LLMs for writing generic, repetitive code, like scaffolding. It's OK with boring, generic stuff. In other words, they're OK in use-cases that programmers need to eliminate , because it means there's high demand for a reusable library, some new syntax sugar, or an improved API.

Boilerplate and plumbing code isn't inherently bad, nor do you improve the codebase by factoring it down to zero with libraries and abstractions. As I've matured as a developer, I've appreciated certain types of boilerplate more and more because it's code that shows up in your git diffs. You don't need to chase down the code in some version of some library to see how something works. Of course, not all boilerplate is…

[deleted]

Re: Making o1, o3, and Sonnet 3.7 hallucinate for everyone

#126

Post by miningape » Sat, Mar 01, 2025, 10:18 PM UTC

immibis wrote: post #119

miningape wrote: post #51

Earlier quoted context omitted.

This, thank you. It pisses me off to no end when people pretend LLMs are smart. They are nothing but a well trained random text generator. Seriously, some these conversations feel like interacting someone who believes casting bones and astrology are accurate. Likely because in both cases they are a result of confirmation bias.

We don't know what smartness is. What if that's what smartness is?

We might not know what it is, but it's not that. At a bare minimum smartness requires abstract reasoning (and no, so-called "reasoning" models do not do that - it's a marketing trick)

The burden of proof for that claim is on you, we cannot start with the assumption these are intelligent systems and disprove it - we have to start with the fact that training is a non-deterministic process and prove that it exhibits intelligence.

Re: Making o1, o3, and Sonnet 3.7 hallucinate for everyone

#127

Post by jwjohnson314 » Sat, Mar 01, 2025, 10:18 PM UTC

The interesting thing here to me is that the llm isn’t ‘hallucinating’, it’s simply regurgitating some data it digested during training.

Re: Making o1, o3, and Sonnet 3.7 hallucinate for everyone

#128

Post by mvdtnz » Sat, Mar 01, 2025, 10:19 PM UTC

jwjohnson314 wrote: post #127

The interesting thing here to me is that the llm isn’t ‘hallucinating’, it’s simply regurgitating some data it digested during training.

What's the difference?

Re: Making o1, o3, and Sonnet 3.7 hallucinate for everyone

#129

Post by mvdtnz » Sat, Mar 01, 2025, 10:21 PM UTC

Terr_ wrote: post #47

kgeist wrote: post #35

Earlier quoted context omitted.

I use LLMs for writing generic, repetitive code, like scaffolding. It's OK with boring, generic stuff. Sure it makes mistakes occasionally but usually it's a no-brainer to fix them.

> I use LLMs for writing generic, repetitive code, like scaffolding. It's OK with boring, generic stuff. In other words, they're OK in use-cases that programmers need to eliminate , because it means there's high demand for a reusable library, some new syntax sugar, or an improved API.

I'll take boring scaffolding code over libraries that perform undebuggable magic with monkey patches, reflection or dynamic code.

Re: Making o1, o3, and Sonnet 3.7 hallucinate for everyone

#130

Post by skissane » Sat, Mar 01, 2025, 10:21 PM UTC

AlotOfReading wrote: post #45

skissane wrote: post #28

Earlier quoted context omitted.

I think a lot of these issues could be avoided if, instead of just a raw model, you have an AI agent which is able to test its own answers against the actual software… it doesn’t matter as much if the model hallucinates if testing weeds out its hallucinations. Sometimes humans “hallucinate” in a similar way - their memory mixes up different programming languages and they’ll try to use syntax from one in another… but…

Testing is better than nothing, but still highly fallible. Take these winning examples from the underhanded C contest [0], [1], where the issues are completely innocuous mistakes that seem to work perfectly despite completely undermining the nominal purpose of the code. You can't substitute an automated process for thinking deeply and carefully about the code. [0] https://www.underhanded-c.org/#winner [1] https://www…

I think it is unlikely (of course not impossible) an LLM would fail in that way.

The underhanded C contest is not a case of people accidentally producing highly misleading code, it is a case of very smart people going to a great amount of effort to intentionally do that.

Most of the time, if your code is wrong, it doesn't work in some obvious way – it doesn't compile, it fails some obvious unit tests, etc.

Code accidentally failing in some subtle way which is easy to miss is a lot rarer – not to say it never happens – but it is the exception not the rule. And it is something humans do too. So if an LLM occasionally does it, they really aren't doing worse than humans are.

> You can't substitute an automated process for thinking deeply and carefully about the code.

Coding LLMs work best when you have an experienced developer checking their output. The LLM focuses on the boring repetitive details leaving the developer more time to look at the big picture – and doing stuff like testing obscure scenarios the LLM probably wouldn't think of.

OTOH, it isn't like all code is equal in terms of consequences if things go wrong. There's a big difference between software processing insurance claims and someone writing a computer game as a hobby. When the stakes are low, lack of experience isn't an issue. We all had to start somewhere.