LLMs, Theory of Mind, and Cheryl's Birthday

    from collections import defaultdict
    
    def find_cheryls_birthday(possible_dates):
        # Parse the dates into month and day
        dates = [date.split() for date in possible_dates]
        months = [month for month, day in dates]
        days = [day for month, day in dates]
    
        # Step 1: Albert knows the month and says he doesn't know the birthday
        # and that Bernard doesn't know either. This implies the month has no unique days.
        month_counts = defaultdict(int)
        day_counts = defaultdict(int)
        for month, day in dates:
            month_counts[month] += 1
            day_counts[day] += 1
    
        # Months with all days appearing more than once
        possible_months = [month for month in month_counts if all(day_counts[day] > 1 for m, day in dates if m == month)]
        filtered_dates = [date for date in dates if date[0] in possible_months]
    
        # Step 2: Bernard knows the day and now knows the birthday
        # This means the day is unique in the filtered dates
        filtered_days = defaultdict(int)
        for month, day in filtered_dates:
            filtered_days[day] += 1
        possible_days = [day for day in filtered_days if filtered_days[day] == 1]
        filtered_dates = [date for date in filtered_dates if date[1] in possible_days]
    
        # Step 3: Albert now knows the birthday, so the month must be unique in remaining dates
        possible_months = defaultdict(int)
        for month, day in filtered_dates:
            possible_months[month] += 1
        final_dates = [date for date in filtered_dates if possible_months[date[0]] == 1]
    
        # Convert back to original format
        return ' '.join(final_dates[0]) if final_dates else "No unique solution found."
    
    # Example usage:
    possible_dates = [
        "May 15", "May 16", "May 19",
        "June 17", "June 18",
        "July 14", "July 16",
        "August 14", "August 15", "August 17"
    ]
    
    birthday = find_cheryls_birthday(possible_dates)
    print(f"Cheryl's Birthday is on {birthday}.")

It's interesting that so many of the model's fail to retrieve this, but any thta do solve it should clearly be able to do so with no reasoning/theory of mind.

This is very interesting and insightful, but I take issue with the above conclusion. Your average software engineer would probably fail to code up a python solution to this problem. But most people would agree that the average software engineer, and the average person, possesses some theory of mind.

This seems to be a pattern I'm noticing with AI. The goalposts keep moving. When I was a kid, the turing test was the holy grail for "artificial intelligence." Now, your run-of-the-mill LLM can breeze through the turing test. But no one seems to care. "They are just imitating us, that doesn't count." Every couple years, AI/ML systems make revolutionary advances, but everyone pretends it's not a big deal because of some new excuse. The latest one being "LLMs can't write a python program to solve an entire class of very challenging logic problems. Therefore LLMs possess no theory of mind."

Let me stick my neck out and say something controversial. Are the latest LLMs as smart as Peter Norvig? No. Are they smarter than your average human? Yes. Can they outperform your average human at a randomly chosen cognitive task that has real-world applications? Yes. This is pretty darn revolutionary. We have crossed the rubicon. We are watching history unfold in real-time.

I agree though, the people who are unable to solve this probably still have a theory of mind. It seems like we're setting a rather high bar.

[0] https://pastebin.com/q33K0HJ1