Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs are mortally terrified of exceptions

twitter.com

Post reply on HN
Print view View original

151–157 of 157 posts

Re: LLMs are mortally terrified of exceptions

#151

Post by chipsrafferty »

bjourne wrote: post #56
karpathy wrote: post #49

Sorry I thought it would be clear and could have clarified that the code itself is just a joke illustrating the point, as an exaggeration. This was the thread if anyone is interested https://chatgpt.com/share/68e82db9-7a28-8007-9a99-bc6f0010d1...

This is stunning English: "Perfect setup for satire. Here’s a Python function that fully commits to the bit — a traumatically over-trained LLM trying to divide numbers while avoiding any conceivable danger:" "Traumatically over-trained", while scoring zero google hits, is an amazingly good description. How can it intuitively know what "traumatic over-training" should mean for LLMs without ever having been taught the…

You need to read more if you think that's stunning English

Re: LLMs are mortally terrified of exceptions

#152

Post by kahnclusions »

muggesmuds wrote: post #92
m463 wrote: post #81

Earlier quoted context omitted.

"Why is a laser beam like goldfish? Because neither one can whistle." - Mike, The Moon is a Harsh Mistress

Fantastic book, just read it. Surprised no movie has been made.

I liked it when I was 17 but have soured on it later after re-reading.

The only reason their libertarian revolution succeeds is because they have a centralised computer that secretly does everything for them.

Re: LLMs are mortally terrified of exceptions

#153

Post by MakeAJiraTicket »

aoeusnth1 wrote: post #146
MakeAJiraTicket wrote: post #86

Earlier quoted context omitted.

Defensive programming is considered "correct" by the people doing the reinforcing, and is a huge part of the corpus that LLM's are trained on. For example, most python code doesn't do manual index management, so when it sees manual index management it is much more likely to freak out and hallucinate a bug. It will randomly promote "silent failure" even when a "silent failure" results in things like infinite loops, be…

You used the word reinforcing, and then asserted there's no reward function. Can you explain how it's possible to perform RL without a reward function, and how the LLM training process maps to that?

LLM actions are divorced from that reward function, it's not something they consult or consider. Reward function in that context doesn't make sense.

Re: LLMs are mortally terrified of exceptions

#154

Post by m463 »

kahnclusions wrote: post #152
muggesmuds wrote: post #92

Earlier quoted context omitted.

Fantastic book, just read it. Surprised no movie has been made.

I liked it when I was 17 but have soured on it later after re-reading. The only reason their libertarian revolution succeeds is because they have a centralised computer that secretly does everything for them.

> I liked it when I was 17

same with pretty much every scifi movie and book from my youth. What movies that wouldn't have been rendered ridiculous by the invention of the cellphone were done in by the hairstyles or fashion.

Re: LLMs are mortally terrified of exceptions

#155

Post by rsynnott »

automatic6131 wrote: post #139
rsynnott wrote: post #110

Earlier quoted context omitted.

The great Verity Stob (unfortunately, in an article which no longer seems to be online, after the Dr Dobbs Journal website finally went away) referred to this behaviour (by _human_ programmers) as "nailing the corpse in an upright position".

https://97-things-every-x-should-know.gitbooks.io/97-things-...

This is her quoting the original article, but as far as I can see the original article is lost to the mists of time. Might be on archive.org, I suppose.

Re: LLMs are mortally terrified of exceptions

#156

Post by gregsadetsky »

keeda wrote: post #96
bspammer wrote: post #76

Earlier quoted context omitted.

I think that’s the funniest joke I’ve ever seen an LLM make. Which probably means it’s copied from somewhere.

If you're an extensive user of ChatGPT, or if you can give it some material about yourself like say, a resume or a LinkedIn profile, ask it to roast you. It will be very specific to the content you give it. Be warned, it can be brutal.

I came back to this comment just to thank you - I started off with Claude, feeding it my personal site, my résumé, the HN roast of me, etc. and it was super funny.

But then, I veered that same conversation into asking for GTM (go to market) advice, and it was actually really good. It actually felt tailored to me (unsurprisingly) and a lot more useful.

As always, I don't know whether this is a very light form of "ai psychosis" haha but still, super grateful for the advice. Cheers

Re: LLMs are mortally terrified of exceptions

#157

Post by bccdee »

bjourne wrote: post #149
bccdee wrote: post #145

Earlier quoted context omitted.

I don't know. It's a classic LLM-ism. "Traumatically over-X" is probably a common enough phrase. The prmpt says, "I don't know what labs are doing to these poor LLMs during RL," so the model connects that to some form of trauma. The training is traumatic, so the model is traumatically over-trained. It sounds fine and flows nicely, but it doesn't quite make sense. Too much training over-fits an LLM; that's not what we…

Eh, you are rationalizing. The phrase "traumatically over-X" is extremely rare. Any problem is easy after you've seen the solution. :) The solution "traumatically over-trained LLM" to the problem "What description best fits karpathy's description?" is certainly not easy to find. Connecting RL, poor LLMs, extreme fear, and welfare to excess training and severe lasting emotional pain is pretty darn impressive. E.g., I…

> The phrase "traumatically over-X" is extremely rare.

There are plenty of "over-x" phrases in English associated with trauma or harm. Do a web search in quotes for "traumatic over{extension/exertion/stimulation}" (off the top of my head) and you'll get direct hits. And this isn't a Markov chain—its doesn't have to pull n-grams directly from its training material. That it could glue trauma and training into "traumatic over-training" is deeply unsurprising to me.

> I couldn't in a million years put it into writing as succinctly and as precisely as the LLM.

If that's the case, then (with respect) that may be down to your skills as a writer. The LLM puts it decently enough, but it's not very expressive and it doesn't add anything.

> Connecting RL, poor LLMs, extreme fear, and welfare to excess training and severe lasting emotional pain is pretty darn impressive

Is it? Really, we're just analogizing it to an abused pet. You over-train your dog, so it gets traumatized. The LLM connects the ideas and then synthesizes a lukewarm sentence to capture that connection at the cost of losing a degree of precision, because LLMs aren't animals. Models are good at those vector-embedding-style conceptual connections—I won't begrudge them that. Expressive use of language and fine-grained reasoning, though? Not so much.

Post reply on HN