Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs are mortally terrified of exceptions

twitter.com

Post reply on HN
Print view View original

21–30 of 157 posts

Re: LLMs are mortally terrified of exceptions

#21

Post by hyperpape »

constantcrying wrote: post #10

If you are dividing two numbers with no prior knowledge of these numbers or any reasonable assumptions you can make and this code is used where you can not rely on the caller to catch an exception and the code is critical for the product, then this is necessary. If you are actually doing safety critical software, e.g. aerospace, medicine or automotive, then this is a good precaution, although you will not be writing…

I'm not sure returning None is any safer than an Exception, because the caller still has to check.

Re: LLMs are mortally terrified of exceptions

#22

Post by isoprophlex »

mewpmewp2 wrote: post #12
constantcrying wrote: post #10

If you are dividing two numbers with no prior knowledge of these numbers or any reasonable assumptions you can make and this code is used where you can not rely on the caller to catch an exception and the code is critical for the product, then this is necessary. If you are actually doing safety critical software, e.g. aerospace, medicine or automotive, then this is a good precaution, although you will not be writing…

I might agree with that, and maybe the example posted by Karpathy is not the greatest, but what I'm constantly being faced with is try catches where it will fail silently or return a fallback/mock response, which essentially means that system will behave unexpectedly in a more subtle way down the line while leaving you clueless to as what the issue was. I have to constantly remind Claude that we want to fail fast.

A good 10% of my Claude.md is yelling at it that no i don't want you to silently handle exceptions six calls deep into the stack and no please don't wrap my return values in weird classes full of dumb status enums "for safety"

Just raise god damn it

Re: LLMs are mortally terrified of exceptions

#23

Post by falcor84 »

That code has many issues, but the one that bothers me the most in practice is this tendency of adding imports inside functions. I can only assume that it's an artifact of them optimizing for a minimal number of edits somewhere in the process, but I expect better.

Re: LLMs are mortally terrified of exceptions

#25

Post by falcor84 »

dijksterhuis wrote: post #18

I mean, the first three cases are just attempting to turn dynamic into static typed... right? maybe just don't aim for uber-safety in a dynamically typed language? :shrugs: (I used to look out for kaparthy's papers ten years ago... i tend to let out an audible sigh when i see his name today)

You shouldn't have the same expectations from a person's tweet as you would from a paper. I don't see any issue with high profile people who are careful in their professional work, putting less thought-through output on social media. At least as long as they don't intentionally/negligently spreading misinformation, which I've never seen Karpathy do.

I for one really enjoy both his longer form work and his shorter takes.

Re: LLMs are mortally terrified of exceptions

#26

Post by freehorse »

metalcrow wrote: post #13

Given that the output describes the function as being done "with extraordinary caution, because you never know what can go wrong", i would guess that the undisclosed prompt was something similar to "generate a division function in python that handles all possible edges cases. be extremely careful". Which seems to say less about LLM training and more about them doing exactly what they are told.

Aside from the absurdity and obvious satirical intention,

1. the code is actually wrong (and is wrong regardless of the absurd exception handling situation)

2. some of the exception handling makes no sense regardless, or is incoherent

3. a less absurd version of this actually happens (edit: commonly in actual irl scenarios) if you put emphasis on exception handling in the prompt

Re: LLMs are mortally terrified of exceptions

#27

Post by nivertech »

recursive wrote: post #19
bobogei81123 wrote: post #15

This is just AI trying to tell us how bad we designed our programming languages to be when exceptions can be thrown pretty much anywhere

So you think java's checked exceptions are a better model? No opinion myself, but that way seems widely considered bad too.

Why do you need exceptions at all? They’re just a different return types in disguise…

Also, division by zero should return Inf

Re: LLMs are mortally terrified of exceptions

#29

Post by falcor84 »

wffurr wrote: post #7

Turns out computer math is actually super hard. Basic operations entail all kinds of undefined behavior and such. This code is a bit verbose but otherwise familiar.

    # Step 3: Preemptively check for catastrophic magnitude differences
    if abs(a) > sys.float_info.max / 2:
        logging.warning("Value of a might cause overflow. Returning infinity just to be sure")
        return math.copysign (float('inf'), a)
    if abs(b)
Does the above code make any sense? I've not worked with this sort of stuff before, but it seems entirely unreasonable to me to check them individually. E.g. if 1 < b < a, then it seems insane to me to return float('inf') for a large but finite a.

Re: LLMs are mortally terrified of exceptions

#30

Post by glitchc »

But what's the prompt that led to this output? Is it just a simple "Write code to divide a by b?" or are there instructions added for code safety or specific behaviours?

I know it's Karpathy, which is why the entire prompt is all the more important to see.

Post reply on HN