I'm sure there is a guy in OpenAI working on the theory of humor and how to make LLMs be comedians. Must be an interesting job.
What do you do for a living?
I teach math how to be funny.
LLMs tell bad jokes because they avoid surprises
61–70 of 163 posts
Re: LLMs tell bad jokes because they avoid surprises
#62Earlier quoted context omitted.
Good completely new jokes are like novel ideas: really hard even for humans. I mean fuck, we have an entire profession dedicated just to making up and telling them, and even theirs don't land half the time.
Exactly. It feels like with LLMs as soon as we achieved the at-the-time astounding breakthrough "LLMs can generate coherent stories" with GPT-2, people have constantly been like "yeah? Well it can't do .". That breakthrough was only 6 years ago! https://openai.com/index/better-language-models/ > We’ve trained a large-scale unsupervised language model which generates coherent paragraphs of text... That was big news. I…
Same thing we saw with game playing:
- It can play chess -> but not at a serious level
- It can beat most people -> but not grandmasters
- It can beat grandmasters -> but it can’t play go
…etc, etc
In a way I guess it’s good that there is always some reason the current version isn’t “really” impressive, as it drives innovation.
But as someone more interested in a holistic understanding of of the world than proving any particular point, it is frustrating to see the goalposts moved without even acknowledging how much work and progress were involved in meeting the goalposts at their previous location.
Re: LLMs tell bad jokes because they avoid surprises
#63I'm hung up on "inevitable in hindsight" part. What is inevitable about, say, dildos getting thrown on the WNBA courts, or Trump saying "everything's computer"?
Those aren't jokes
Re: LLMs tell bad jokes because they avoid surprises
#64So I just tried with ChatGPT, with the prompt at bottom, borrowing the description of good joke from the article. I think there's some interesting stuff, even with this minimal prompting. The example below was from down the line, ChatGPT kept on offering jokes with different style. Man: “Why do you always bury bones in the garden?”, Dog: “Because the bank keeps asking for ID.” Man: “Don’t beg at the table.”, Dog: “Do…
“Don’t eat in my begging spot.” is pretty good.
Re: LLMs tell bad jokes because they avoid surprises
#65I'm sure there is a guy in OpenAI working on the theory of humor and how to make LLMs be comedians. Must be an interesting job.
In the pre-LLM days a friend's lab worked on a joke detector for The New Yorker. One measure they used was trigram surprise. Roughly P(AB) + P(BC) >> P(ABC). For example, "alleged killer" and "killer whale" are both common, but "alleged killer whale" is surprising.
That reminds me of a joke I liked from Tim Heidecker when he was ribbing Maynard Keenan about his wine making:
"The blood of Christ is essentially wine, correct?"
Yes.
"Who are you to put that in a bottle?"
So a logical spoke can be inferred as well, blood->wine wine->bottle blood->bottle. That uses their own logical inferences against them as a "trick" which is another funny element for people. Using that to vault straight to the punch line makes the joke better, but you have to be sure the audience is on board, which is why there is a bit of reinforcement at the beginning of the joke to force them onboard.
Re: LLMs tell bad jokes because they avoid surprises
#66I'm sure there is a guy in OpenAI working on the theory of humor and how to make LLMs be comedians. Must be an interesting job.
I have no doubt plenty of smart engineers at tech companies would rather reinvent the wheel than read a book on theatre. But if anyone’s interested, there are plenty of great books talking about the philosophy of comedy, and why some things work on stage and some don’t. I highly recommend Keith Johnstone’s “Impro”. He’s the guy who invented modern improv comedy and theatre sports.
He says things are funny if they’re obvious. But not just any obvious. They have to be something in the cloud of expectation of the audience. Like, something they kinda already thought but hadn’t named. If you have a scene where someone’s talking to a frog about love, it’s not funny for the talking frog to suddenly go to space. But it might be funny to ask the frog why it can talk. Or ask about gossip in the royal palace. Or say “if you’re such a catch, how’d you end up as a frog?”.
If good comedy is obvious, you’d think LLMs would be good at it. Honestly I think LLMs fall down by not being specific enough in detail. They don’t have ideas and commit to them. They’re too bland. Maybe their obvious just isn’t the same as ours.
Re: LLMs tell bad jokes because they avoid surprises
#67Earlier quoted context omitted.
Initially impressed. But Googling for that shows a lot of hits.
Absolutely, it's an old joke. But the fact that the thing even knew it was a joke and told it reasonably well... the bar is on the floor. (the one the guy in the joke walked into)
It's one of several canned jokes you get with a very high likelihood. Try a couple of times and be dismayed - you'll be getting the same three jokes over and over again. I'm fairly certain they were specifically tuned to return that. It takes special prompting to get it to write a new joke, and the results are typically disastrous / surreal.
Re: LLMs tell bad jokes because they avoid surprises
#68Earlier quoted context omitted.
Exactly. It feels like with LLMs as soon as we achieved the at-the-time astounding breakthrough "LLMs can generate coherent stories" with GPT-2, people have constantly been like "yeah? Well it can't do .". That breakthrough was only 6 years ago! https://openai.com/index/better-language-models/ > We’ve trained a large-scale unsupervised language model which generates coherent paragraphs of text... That was big news. I…
Same thing we saw with game playing: - It can play chess -> but not at a serious level - It can beat most people -> but not grandmasters - It can beat grandmasters -> but it can’t play go …etc, etc In a way I guess it’s good that there is always some reason the current version isn’t “really” impressive, as it drives innovation. But as someone more interested in a holistic understanding of of the world than proving an…
> it is frustrating to see the goalposts moved without even acknowledging how much work and progress were involved in meeting the goalposts at their previous location.
Half the HN front page for the past years has been nothing but acknowledging the progress of LLMs in sundry ways. I wish we actually stopped for a second. It’s all people seem to want to talk about anymore.
Re: LLMs tell bad jokes because they avoid surprises
#69Earlier quoted context omitted.
Same thing we saw with game playing: - It can play chess -> but not at a serious level - It can beat most people -> but not grandmasters - It can beat grandmasters -> but it can’t play go …etc, etc In a way I guess it’s good that there is always some reason the current version isn’t “really” impressive, as it drives innovation. But as someone more interested in a holistic understanding of of the world than proving an…
> it is frustrating to see the goalposts moved without even acknowledging how much work and progress were involved in meeting the goalposts at their previous location. Half the HN front page for the past years has been nothing but acknowledging the progress of LLMs in sundry ways. I wish we actually stopped for a second. It’s all people seem to want to talk about anymore.
I should have been more clear. Let me rephrase as: among those who dismiss the latest innovations as nothing special because there is still further to go, it would be nice to acknowledgment when goalposts are moved.
Re: LLMs tell bad jokes because they avoid surprises
#70This sounds really convincing but I'm not sure it's actually correct. The author is conflating the surprise of punchlines with their likelihood. To put it another way, ask a professional comedian to complete a joke with a punchline. It's very likely that they'll give you a funny surprising answer. I think the real explanation is that good jokes are actually extremely difficult. I have young children (4 and 6). Even 6…
And he's too generous towards human intelligence.
Good stories and good jokes DO follow predictable patterns.