Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs are mortally terrified of exceptions

twitter.com

Post reply on HN
Print view View original

51–60 of 157 posts

Re: LLMs are mortally terrified of exceptions

#51

Post by nivertech »

dmoy wrote: post #48
nivertech wrote: post #45

Earlier quoted context omitted.

Why not just to use IEEE 754? > According to the IEEE 754 standard, floating-point division by zero is not an error but results in special values: positive infinity, negative infinity, or Not a Number (NaN). The specific result depends on the numerator

Because sometimes it's very wrong Way back when during my EE course days, we had like a whole semester devoted to weird edge cases like this, and spent month on ieee754 (precision loss, Nan, divide by zero, etc) When you took an ieee754 divide by zero value as gospel and put it in the context of a voltage divisor that is always negative or zero, getting a positive infinity value out of divide by zero was very wrong,…

With IEEE 754 you can always explicitly check for edge cases.

But with exceptions you can’t use SIMD / vectorization.

Re: LLMs are mortally terrified of exceptions

#52

Post by chis »

karpathy wrote: post #49

Sorry I thought it would be clear and could have clarified that the code itself is just a joke illustrating the point, as an exaggeration. This was the thread if anyone is interested https://chatgpt.com/share/68e82db9-7a28-8007-9a99-bc6f0010d1...

I think there’s always a danger of these foundational model companies doing RLHF on non-expert users, and this feels like a case of that.

The AIs in general feel really focused on making the user happy - your example, and another one is how they love adding emojis to the stout and over-commenting simple code.

Re: LLMs are mortally terrified of exceptions

#53

Post by Terr_ »

recursive wrote: post #19
bobogei81123 wrote: post #15

This is just AI trying to tell us how bad we designed our programming languages to be when exceptions can be thrown pretty much anywhere

So you think java's checked exceptions are a better model? No opinion myself, but that way seems widely considered bad too.

> So you think java's checked exceptions are a better model?

Checked Exceptions are a good concept which just needed more syntactic-sugar. (Like easily specifying that one kind of exception should be wrapped into another.) The badness is not in the logic but in the ecology, the ways that junior/lazy developers are incentivized to take horrible shortcuts.

Checked exceptions are fundamentally the same as managing the types of return-values... except the language doesn't permit the same horrible-shortcuts for people to abuse.

Meme reaction: http://imgur.com/iYE5nLA

_____

Prior discussion: https://news.ycombinator.com/item?id=42946597

Re: LLMs are mortally terrified of exceptions

#54

Post by dmoy »

nivertech wrote: post #51
dmoy wrote: post #48

Earlier quoted context omitted.

Because sometimes it's very wrong Way back when during my EE course days, we had like a whole semester devoted to weird edge cases like this, and spent month on ieee754 (precision loss, Nan, divide by zero, etc) When you took an ieee754 divide by zero value as gospel and put it in the context of a voltage divisor that is always negative or zero, getting a positive infinity value out of divide by zero was very wrong,…

With IEEE 754 you can always explicitly check for edge cases. But with exceptions you can’t use SIMD / vectorization.

Yea that's totally fair, you'd need to build it in as a first class behavior of your code, doesn't necessarily mean that exceptions is the right way to do it.

Re: LLMs are mortally terrified of exceptions

#55

Post by why_at »

karpathy wrote: post #49

Sorry I thought it would be clear and could have clarified that the code itself is just a joke illustrating the point, as an exaggeration. This was the thread if anyone is interested https://chatgpt.com/share/68e82db9-7a28-8007-9a99-bc6f0010d1...

This part from the first try made me laugh:

      if random.random()

Re: LLMs are mortally terrified of exceptions

#56

Post by bjourne »

karpathy wrote: post #49

Sorry I thought it would be clear and could have clarified that the code itself is just a joke illustrating the point, as an exaggeration. This was the thread if anyone is interested https://chatgpt.com/share/68e82db9-7a28-8007-9a99-bc6f0010d1...

This is stunning English: "Perfect setup for satire. Here’s a Python function that fully commits to the bit — a traumatically over-trained LLM trying to divide numbers while avoiding any conceivable danger:" "Traumatically over-trained", while scoring zero google hits, is an amazingly good description. How can it intuitively know what "traumatic over-training" should mean for LLMs without ever having been taught the concept?

Re: LLMs are mortally terrified of exceptions

#57

Post by cma »

chis wrote: post #52
karpathy wrote: post #49

Sorry I thought it would be clear and could have clarified that the code itself is just a joke illustrating the point, as an exaggeration. This was the thread if anyone is interested https://chatgpt.com/share/68e82db9-7a28-8007-9a99-bc6f0010d1...

I think there’s always a danger of these foundational model companies doing RLHF on non-expert users, and this feels like a case of that. The AIs in general feel really focused on making the user happy - your example, and another one is how they love adding emojis to the stout and over-commenting simple code.

And more advanced users are more likely to opt out of training on their data, Google gets around it with a free api period where you can't opt out and I think from did some of that too, through partnerships with tool companies, but not sure if you can ever opt out there.

Re: LLMs are mortally terrified of exceptions

#58

Post by dragonwriter »

nivertech wrote: post #27
recursive wrote: post #19

Earlier quoted context omitted.

So you think java's checked exceptions are a better model? No opinion myself, but that way seems widely considered bad too.

Why do you need exceptions at all? They’re just a different return types in disguise… Also, division by zero should return Inf

> Why do you need exceptions at all? They’re just a different return types in disguise…

You don’t need exceptions, and they can be replaced by more intricate return types.

OTOH, for the intended use case for signalling conditions that most code directly calling a function does not expect and cannot do anything about, unchecked exceptions reduce code clutter (checked exceptions are isomorphic to "more intricate return types"), at the expense of making the potential error cases less visible.

Whether this tradeoff is a net benefit is somewhat subjective and, IMO, highly situational. but if (unchecked) exceptions are available, you can always convert any encountered in your code into return values by way of handlers (and conversely you can also do the opposite), whereas if they aren’t available, you have no choice.

Re: LLMs are mortally terrified of exceptions

#59

Post by drekipus »

bjourne wrote: post #56
karpathy wrote: post #49

Sorry I thought it would be clear and could have clarified that the code itself is just a joke illustrating the point, as an exaggeration. This was the thread if anyone is interested https://chatgpt.com/share/68e82db9-7a28-8007-9a99-bc6f0010d1...

This is stunning English: "Perfect setup for satire. Here’s a Python function that fully commits to the bit — a traumatically over-trained LLM trying to divide numbers while avoiding any conceivable danger:" "Traumatically over-trained", while scoring zero google hits, is an amazingly good description. How can it intuitively know what "traumatic over-training" should mean for LLMs without ever having been taught the…

The same way that you and I think up a word and what it might mean without being taught the concept.

Adverb + verb

Post reply on HN