Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs, Theory of Mind, and Cheryl's Birthday

github.com

Post reply on HN
Print view View original

81–90 of 150 posts

Re: LLMs, Theory of Mind, and Cheryl's Birthday

#81

Post by Terretta »

lagmg05 wrote: post #71
oli5679 wrote: post #70

Gp1-o1 preview solves this puzzle correctly in 13 seconds and has a thorough logical deduction in the comments and explanation. I think it’s a bit unfair on llm to ask it to retrieve the puzzle definition from its training data. I posted the info on the puzzle from his notebook. https://chatgpt.com/share/670103ae-1c18-8011-8068-dd21793727...

The question is if it solved the puzzle correctly before Norvig's article appeared. It could have been trained (I am told that existing models can be modified and augmented in any Llama discussion) on the article or on HN comments. There could even be an added routine that special cases trick questions and high profile criticisms.

> The question is if it solved the puzzle correctly before Norvig's article appeared. It could have been trained...

This caught me by surprise — is there a suggestion or evidence that despite the "knowledge cutoff" OpenAI is continuously retraining GPT-4o's chat-backing model(s) on day over day updates to the web?

Re: LLMs, Theory of Mind, and Cheryl's Birthday

#82

Post by aithrowawaycomm »

IanCal wrote: post #65

I'm not a huge fan of using these kind of riddles or gotchas. Other comments have riddle variants which feel also like ways of tripping someone up - if you don't spot the change you fail. And what's more the originals are things that lots of people struggle with (that's why they're riddles not basic questions). There's also little prompting, which feels like calling up a random person and demanding they solve a riddl…

The more dangerous - and far more ridiculous - assumption is that if the LLM does write the correct code then it does understand what people know. I am amazed at how common that view is here.

Re: LLMs, Theory of Mind, and Cheryl's Birthday

#83

Post by aithrowawaycomm »

zmgsabst wrote: post #51
emseetech wrote: post #41

Earlier quoted context omitted.

I would venture to guess that it has value in that we couldn't even begin to properly test a theory of mind if they can't yet reason through a logic puzzle. It would be like testing if a society could build a rocket ship when they don't know how to use tools.

What animals have theory of mind?

Ravens do: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1559847/

I don't have a link but I believe squirrels do, for the exact same reason: their lives are full of intense intra-species competition, with other squirrels constantly raiding each others' stashes. IIRC squirrels will even deceptively hide acorns if they know another squirrel is watching.

Re: LLMs, Theory of Mind, and Cheryl's Birthday

#84

Post by mirekrusin »

samrus wrote: post #75
mirekrusin wrote: post #69

Earlier quoted context omitted.

Not to take anything from what you're saying but any person involved in hiring process _for programmers_ will agree that people struggle with trivia far easier than this example.

Yeah but an LLM can't be excused being nervous or not tired or just not operating at its best like a human can. They are always operating at their best, so if they trip up once, that's their limit

Yes, but they will only keep getting better; whether we like it or not sooner or later we’re going to be hiring api keys, not people.

Re: LLMs, Theory of Mind, and Cheryl's Birthday

#85

Post by ifdefdebug »

IanCal wrote: post #65

I'm not a huge fan of using these kind of riddles or gotchas. Other comments have riddle variants which feel also like ways of tripping someone up - if you don't spot the change you fail. And what's more the originals are things that lots of people struggle with (that's why they're riddles not basic questions). There's also little prompting, which feels like calling up a random person and demanding they solve a riddl…

> if you don't spot the change you fail

But that's exactly how real world works too.

Re: LLMs, Theory of Mind, and Cheryl's Birthday

#86

Post by mark_l_watson »

Nice! I use various LLMs many times a day as a limited coding tool and something to bounce ideas off of, and it is impossible to not think about how LLMs work and what their limitations are.

I tried just asking Claude Sonet to solve the Cheryl’s Birthday word problem, changing the dates. Pretty cool that it can solve it as a word problem, and LLMs will keep getting better at coding.

As a slight tangent: I used a combination of Gemini, GPT-4o, and Claude last week to write Common Lisp code for a simple RDF data store and the subset of SPARQL queries that I thought I would need in embedded Common Lisp applications. This process was far from automatic: I initially provided almost two pages of English instructions, and I had to help debug non-working code by adding debug statements and then show the models the code with print statements and the new output. I also did the optional thing of asking for stylistic changes. TLDR: saved me time and I liked the final code.

I always enjoy it when people like Peter and Karpathy write relatively simple code to share ideas. I am a fairly good coder (I had the meaningless title Master Software Engineer at Capital One) but I like to read other people’s code, and I must admit that I spend more time reading code on GitHub than I spend reading technical papers.

Re: LLMs, Theory of Mind, and Cheryl's Birthday

#87

Post by Fripplebubby »

lagmg05 wrote: post #71
oli5679 wrote: post #70

Gp1-o1 preview solves this puzzle correctly in 13 seconds and has a thorough logical deduction in the comments and explanation. I think it’s a bit unfair on llm to ask it to retrieve the puzzle definition from its training data. I posted the info on the puzzle from his notebook. https://chatgpt.com/share/670103ae-1c18-8011-8068-dd21793727...

The question is if it solved the puzzle correctly before Norvig's article appeared. It could have been trained (I am told that existing models can be modified and augmented in any Llama discussion) on the article or on HN comments. There could even be an added routine that special cases trick questions and high profile criticisms.

While this is technically possible, it is not remotely practical and the downside risk of pushing out a borked model is much higher than the upside.

Training the model is expensive (obviously), but even if you are only training it slightly, running evaluations to determine whether the particular training checkpoint is at or above the quality bar is expensive, too.

Re: LLMs, Theory of Mind, and Cheryl's Birthday

#88

Post by Fripplebubby »

ifdefdebug wrote: post #85
IanCal wrote: post #65

I'm not a huge fan of using these kind of riddles or gotchas. Other comments have riddle variants which feel also like ways of tripping someone up - if you don't spot the change you fail. And what's more the originals are things that lots of people struggle with (that's why they're riddles not basic questions). There's also little prompting, which feels like calling up a random person and demanding they solve a riddl…

> if you don't spot the change you fail But that's exactly how real world works too.

This has not been my experience with the real world. Riddles and gotchas have played a very small role, so far.

Re: LLMs, Theory of Mind, and Cheryl's Birthday

#89

Post by theptip »

ifdefdebug wrote: post #85
IanCal wrote: post #65

I'm not a huge fan of using these kind of riddles or gotchas. Other comments have riddle variants which feel also like ways of tripping someone up - if you don't spot the change you fail. And what's more the originals are things that lots of people struggle with (that's why they're riddles not basic questions). There's also little prompting, which feels like calling up a random person and demanding they solve a riddl…

> if you don't spot the change you fail But that's exactly how real world works too.

Most people with theory of mind can’t trivially solve this problem though. So the test doesn’t disprove ToM in general, just that it memorizes some results.
Post reply on HN