Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs, Theory of Mind, and Cheryl's Birthday

github.com

Post reply on HN
Print view View original

61–70 of 150 posts

Re: LLMs, Theory of Mind, and Cheryl's Birthday

#61

Post by [deleted] »

AdieuToLogic wrote: post #49
aithrowawaycomm wrote: post #26

AI researchers need to learn what terms like "theory of mind" actually mean before they write dumb crap like this. Theory of mind is about attributing mental states to others, not information. What Norvig has done here is present a logic puzzle, one that works equally well when the agents are Prolog programs instead of clever children. There's no "mind" in this puzzle at all. Norvig is being childishly ignorant to ca…

> AI researchers need to learn what terms like "theory of mind" actually mean before they write dumb crap like this. > Of course he is hardly the only offender: arrogant disregard for psychology is astonishingly common among LLM researchers. Maybe they should turn off ChatGPT and read a book. Perhaps this[0] book? It is one of the seminal texts on AI topics. Maybe Norvig should read it. 0 - http://aima.cs.berkeley.ed…

[deleted]

Re: LLMs, Theory of Mind, and Cheryl's Birthday

#62

Post by skybrian »

sdenton4 wrote: post #57
skybrian wrote: post #56

Earlier quoted context omitted.

It did look that way and it's a fun way to interpret it, but pattern-matching on a pretty obvious pattern in the text (several failed fixes in a row) seems more likely. LLM's will repeat patterns in other circumstances too.

I mean, humans do this too... If I tell an interviewee that they've done something wrong a few times, they'll have less confidence going forward (unless they're a sociopath), and typically start checking their work more closely to preempt problems. This particular instance of in-context pattern matching doesn't seem obviously unintelligent to me.

This was code that finished successfully (no stack trace) and rendered an image, but the output didn't match what I asked it to do, so I told it what it actually looked like. Code Interpreter couldn't check its work in that case, because it couldn't see it. It had to rely on me to tell it.

So it was definitely writing "here's the answer... that failed, let's try again" without checking its work, because it never prompted me. You could call that "hallucinating" a failure.

I also found that it "hallucinated" other test results - I'd ask it to write some code that printed a number to the console and told it what the number was supposed to be, and then it would say it "worked," reporting the expected value instead of the actual number.

I also asked it to write a test and run it, and it would say it passed, and I'd look at the actual output and it failed.

So, asking it to write tests didn't work as well as I'd hoped; it often "sees" things based on what would complete the pattern instead of the actual result.

Re: LLMs, Theory of Mind, and Cheryl's Birthday

#64

Post by xelxebar »

jawns wrote: post #9

A long time ago, I created a version of this challenge called "Cheryl's Murder." My notebook not only solves logical induction problems like "Cheryl's Birthday," but it also generates them. https://github.com/shaungallagher/cheryls-murder/blob/master...

> Twice a year, all of the software engineers at our company are given several days to work on a project of their choosing.

Monetate sounds like it has (had?) some interesting leadership!

Re: LLMs, Theory of Mind, and Cheryl's Birthday

#65

Post by IanCal »

I'm not a huge fan of using these kind of riddles or gotchas. Other comments have riddle variants which feel also like ways of tripping someone up - if you don't spot the change you fail. And what's more the originals are things that lots of people struggle with (that's why they're riddles not basic questions).

There's also little prompting, which feels like calling up a random person and demanding they solve a riddle straight away without talking it through.

Part of the assumption here is that if llms don't write the right code they don't understand what people know. I'd wager that a huge number of people couldn't solve this puzzle yet fully understand what different people have their own internal thoughts and knowledge.

Re: LLMs, Theory of Mind, and Cheryl's Birthday

#66

Post by isaacfrond »

erwald wrote: post #2

o1 mini seems to get it on the first try (I didn't vet the code, but I tested it and it works on both examples provided in the notebook, `dates` and `gabe_dates`): from collections import defaultdict def find_cheryls_birthday(possible_dates): # Parse the dates into month and day dates = [date.split() for date in possible_dates] months = [month for month, day in dates] days = [day for month, day in dates] # Step 1: Al…

despite the name ‘mini’. it is actually more optimized for code. so that makes sense.

Re: LLMs, Theory of Mind, and Cheryl's Birthday

#67

Post by rghall102 »

jfcoa wrote: post #4

This seems like a terrible test case since python examples are readily available in the training data: https://rosettacode.org/wiki/Cheryl%27s_birthday It's interesting that so many of the model's fail to retrieve this, but any thta do solve it should clearly be able to do so with no reasoning/theory of mind.

It is fascinating that the R solution just below the Python solution is much shorter and more readable. The same applies to Ruby and various Lisps.

It even applies to the VisualBasic solution!

Re: LLMs, Theory of Mind, and Cheryl's Birthday

#68

Post by rtlacn »

IanCal wrote: post #65

I'm not a huge fan of using these kind of riddles or gotchas. Other comments have riddle variants which feel also like ways of tripping someone up - if you don't spot the change you fail. And what's more the originals are things that lots of people struggle with (that's why they're riddles not basic questions). There's also little prompting, which feels like calling up a random person and demanding they solve a riddl…

Humans can solve these riddles even if they have never seen a similar one.

LLMs need to have seen at least a similar riddle with the same structure and/or code to plagiarize.

Humans can deal with modified riddles if they are alert and awake. An LLM is always alert and awake, so it should be able to handle modified riddles, but it cannot.

Re: LLMs, Theory of Mind, and Cheryl's Birthday

#69

Post by mirekrusin »

rtlacn wrote: post #68
IanCal wrote: post #65

I'm not a huge fan of using these kind of riddles or gotchas. Other comments have riddle variants which feel also like ways of tripping someone up - if you don't spot the change you fail. And what's more the originals are things that lots of people struggle with (that's why they're riddles not basic questions). There's also little prompting, which feels like calling up a random person and demanding they solve a riddl…

Humans can solve these riddles even if they have never seen a similar one. LLMs need to have seen at least a similar riddle with the same structure and/or code to plagiarize. Humans can deal with modified riddles if they are alert and awake. An LLM is always alert and awake, so it should be able to handle modified riddles, but it cannot.

Not to take anything from what you're saying but any person involved in hiring process _for programmers_ will agree that people struggle with trivia far easier than this example.

Re: LLMs, Theory of Mind, and Cheryl's Birthday

#70

Post by oli5679 »

Gp1-o1 preview solves this puzzle correctly in 13 seconds and has a thorough logical deduction in the comments and explanation.

I think it’s a bit unfair on llm to ask it to retrieve the puzzle definition from its training data. I posted the info on the puzzle from his notebook.

https://chatgpt.com/share/670103ae-1c18-8011-8068-dd21793727...

Post reply on HN