but then, why code with exceptions, why not perform pre-flight/pre-validation checks and minimize exceptions to the truly unknown?
LLMs are mortally terrified of exceptions
31–40 of 157 posts
Re: LLMs are mortally terrified of exceptions
#32Turns out computer math is actually super hard. Basic operations entail all kinds of undefined behavior and such. This code is a bit verbose but otherwise familiar.
# Step 3: Preemptively check for catastrophic magnitude differences if abs(a) > sys.float_info.max / 2: logging.warning("Value of a might cause overflow. Returning infinity just to be sure") return math.copysign (float('inf'), a) if abs(b) Does the above code make any sense? I've not worked with this sort of stuff before, but it seems entirely unreasonable to me to check them individually. E.g. if 1 < b < a, then it…
Ignoring the sign of b for big a can't be right.
Re: LLMs are mortally terrified of exceptions
#33Earlier quoted context omitted.
So you think java's checked exceptions are a better model? No opinion myself, but that way seems widely considered bad too.
Why do you need exceptions at all? They’re just a different return types in disguise… Also, division by zero should return Inf
Or -Inf, depending on the sign of the zero, which might catch some programmers by surprise, but is of course the correct thing to do.
Re: LLMs are mortally terrified of exceptions
#34Earlier quoted context omitted.
So you think java's checked exceptions are a better model? No opinion myself, but that way seems widely considered bad too.
Why do you need exceptions at all? They’re just a different return types in disguise… Also, division by zero should return Inf
Unchecked exceptions are more like a shutdown event, which can be intercepted at any point along the call stack, which is useful and not like a return type.
Re: LLMs are mortally terrified of exceptions
#35But what's the prompt that led to this output? Is it just a simple "Write code to divide a by b?" or are there instructions added for code safety or specific behaviours? I know it's Karpathy, which is why the entire prompt is all the more important to see.
"Write me a code that divides a by b and make sure it is safe and handles all edge cases"[1] or something and some languages have more than others.
[1] Probably with some "make you sure handle ALL cases in existence", or emphasis, along those lines.
Re: LLMs are mortally terrified of exceptions
#36Earlier quoted context omitted.
Why do you need exceptions at all? They’re just a different return types in disguise… Also, division by zero should return Inf
Unchecked exceptions are more like a shutdown event, which can be intercepted at any point along the call stack, which is useful and not like a return type.
Why do you need the call stack at all?
Re: LLMs are mortally terrified of exceptions
#37Earlier quoted context omitted.
Why do you need exceptions at all? They’re just a different return types in disguise… Also, division by zero should return Inf
Or -Inf, depending on the sign of the zero, which might catch some programmers by surprise, but is of course the correct thing to do.
a/0 = Inf when a>0
a/0 = -Inf when aRe: LLMs are mortally terrified of exceptions
#38I've noted that LLMs tend to produce defensive code to a fault. Lots of unnecessary checks, e.g. check for null/None/undefined multiple times for same valie. This can lead to really hard to read code, even for the LLM itself.
The RL objectives probably heavily penalize exceptions, but don't reward much for code readability or simplicity.
Re: LLMs are mortally terrified of exceptions
#39That code has many issues, but the one that bothers me the most in practice is this tendency of adding imports inside functions. I can only assume that it's an artifact of them optimizing for a minimal number of edits somewhere in the process, but I expect better.
It's to make imports lazy, to solve the issue of slow import at startup.
Re: LLMs are mortally terrified of exceptions
#40Given that the output describes the function as being done "with extraordinary caution, because you never know what can go wrong", i would guess that the undisclosed prompt was something similar to "generate a division function in python that handles all possible edges cases. be extremely careful". Which seems to say less about LLM training and more about them doing exactly what they are told.
I interpreted the function code as being a deliberately exaggerated satirical example that was illustrative of the experience he was having. So yes, in that example it was probably told to be overly cautious, but I agree with him that the default of LLMs seems to be a bit more cautious than I would like.