Live data from Hacker News

LLMs are mortally terrified of exceptions

twitter.com

131–140 of 157 posts

Re: LLMs are mortally terrified of exceptions

#131

Earlier quoted context omitted.

Periodic reminder that there’s also HN Wrapped. [0] [0]: https://hn-wrapped.kadoa.com

ooooh boy, gotta mentally prepare myself for this one damn these ai's are good!

"The user will start a comment with 'I'm a social libertarian but...' only to be immediately downvoted by both libertarians and socialists. The irony will not be lost on them, just everyone else."

I can't say I'm not impressed. That's very funny

Re: LLMs are mortally terrified of exceptions

#132
post #55

Sorry I thought it would be clear and could have clarified that the code itself is just a joke illustrating the point, as an exaggeration. This was the thread if anyone is interested https://chatgpt.com/share/68e82db9-7a28-8007-9a99-bc6f0010d1...

This part from the first try made me laugh: if random.random()

Years and years ago, the MongoDB Java driver had something like this to skip logging sometimes in one of its error handling routines.

   } catch (Exception e) {
                if (!((_ok) ? true : (Math.random() > 0.1))) {
                    return res;
                }

                final StringBuilder logError = (new StringBuilder("Server seen down: ")).append(_addr);

                /* edited for brevity: log the error */
 
https://github.com/mongodb/mongo-java-driver/blob/1d2e6faa80...

Re: LLMs are mortally terrified of exceptions

#133
post #48

Earlier quoted context omitted.

Because sometimes it's very wrong Way back when during my EE course days, we had like a whole semester devoted to weird edge cases like this, and spent month on ieee754 (precision loss, Nan, divide by zero, etc) When you took an ieee754 divide by zero value as gospel and put it in the context of a voltage divisor that is always negative or zero, getting a positive infinity value out of divide by zero was very wrong,…

But IEEE 754 works as you described in your last comment. It doesn't take the numerator's sign. So what's wrong? Can you give more context on your voltage math? Was the numerator sometimes negative? If the problem is that your divisor calculation sometimes resulted in positive zero, that doesn't sound like the standard being wrong without more info.

> But IEEE 754 works as you described in your last comment. It doesn't take the numerator's sign. So what's wrong?

The numerator was always positive. The denominator was always negative (negative voltage is a pretty common thing), except when it became zero. That led to surprising behavior.

Right the whole point of the exercise was that sometimes the standard is wrong for your specific problem at hand. We spent lecture after lecture going over exactly how ieee754 precision loss worked, and other edge cases, so we could know how to exactly follow the standard.

Then we had an example where the sudden sign flip from a/-0.00000000001 = to a/0 = would cause big problems with a calculation. If you didn't explicitly handle the divide by zero case and do the "correct for domain, but not following ieee754 standard" way, then you'd fry a component.

It's been a long time so I don't remember the exact setup, just the higher level lesson of "don't blindly follow standards and assume you don't need to check edge cases (exception or otherwise) because the standard does things a certain way".

Re: LLMs are mortally terrified of exceptions

#134
post #96

Earlier quoted context omitted.

If you're an extensive user of ChatGPT, or if you can give it some material about yourself like say, a resume or a LinkedIn profile, ask it to roast you. It will be very specific to the content you give it. Be warned, it can be brutal.

Periodic reminder that there’s also HN Wrapped. [0] [0]: https://hn-wrapped.kadoa.com

Spot on and I don't even mind.

Re: LLMs are mortally terrified of exceptions

#135

Earlier quoted context omitted.

So you think java's checked exceptions are a better model? No opinion myself, but that way seems widely considered bad too.

Why do you need exceptions at all? They’re just a different return types in disguise… Also, division by zero should return Inf

Division by zero is mathematically undefined. So two's complement integer division by zero is always undefined.

For floating point there is the interesting property that 0 is signed due to its signed magnitude representation. Mathematically 0 is not signed but in floating point signed magnitude representation, "+0" is equivalent to lim x->0+ x and "-0" is equivalent to lim x->0- x.

This is the only situation where a floating point division by "zero" makes mathematical sense, where a finite number divided by a signed zero will return a signed +/-Inf, and a 0/0 will return a NaN.

Why should 0/0 return a NaN instead of Inf? Because lim x->0 4x/x = 4, NOT Inf.

Re: LLMs are mortally terrified of exceptions

#136
Agree that LLMs go too far on error catching..

BUT, to play devil's advocate a little: Most human coders should be writing a lot more try/catch blocks than they actually do. It's very common that you don't actually want an error in one section (however unlikely) to interrupt the overall operation. (and sometimes you do, it just depends)

Re: LLMs are mortally terrified of exceptions

#137
post #55

Earlier quoted context omitted.

This part from the first try made me laugh: if random.random()

Years and years ago, the MongoDB Java driver had something like this to skip logging sometimes in one of its error handling routines. } catch (Exception e) { if (!((_ok) ? true : (Math.random() > 0.1))) { return res; } final StringBuilder logError = (new StringBuilder("Server seen down: ")).append(_addr); /* edited for brevity: log the error */ https://github.com/mongodb/mongo-java-driver/blob/1d2e6faa80...

One of my earlier jobs a decade ago involved doing pipeline development and Jenkins administration for the on-site developer lab on one of the NRO projects, and I inserted a random build failure code snippet to test that pipelines could recover from builds that failed for unpredictable reasons, like a network error rather than anything actually wrong with the build. I had to do this on the real system because we didn't have funds for a staging environment for the dev environment, and naturally I forgot to get rid of it when I was done. So builds randomly failed for years after that before I remembered and fixed it.

Re: LLMs are mortally terrified of exceptions

#138
post #55

Earlier quoted context omitted.

This part from the first try made me laugh: if random.random()

Years and years ago, the MongoDB Java driver had something like this to skip logging sometimes in one of its error handling routines. } catch (Exception e) { if (!((_ok) ? true : (Math.random() > 0.1))) { return res; } final StringBuilder logError = (new StringBuilder("Server seen down: ")).append(_addr); /* edited for brevity: log the error */ https://github.com/mongodb/mongo-java-driver/blob/1d2e6faa80...

If we’re talking about funny error msgs, a buddy of mine got this yesterday in salesforce. It’s not _that_ funny but pretty funny for Salesforce.

System.DmlException: Insert failed. First exception on row 0; first error: UNKNOWN_EXCEPTION, Something is very wrong: []

Re: LLMs are mortally terrified of exceptions

#139
post #46

This is a parody but the phenomenon is real. My uninformed suspicion is that this kind of defensive programming somehow improves performance during RLVR. Perhaps the model sometimes comes up with programs that are buggy enough to emit exceptions, but close enough to correct that they produce the right answer after swallowing the exceptions. So the model learns that swallowing exceptions sometimes improves its reward.…

The great Verity Stob (unfortunately, in an article which no longer seems to be online, after the Dr Dobbs Journal website finally went away) referred to this behaviour (by _human_ programmers) as "nailing the corpse in an upright position".

https://97-things-every-x-should-know.gitbooks.io/97-things-...
Post reply on HN