LLMs, Theory of Mind, and Cheryl's Birthday

Script generated by o1-preview:

  ```python
  # List of possible dates
  dates = [
      ('May', 15), ('May', 16), ('May', 19),
      ('June', 17), ('June', 18),
      ('July', 14), ('July', 16),
      ('August', 14), ('August', 15), ('August', 17)
  ]
  
  def solve_cheryls_birthday(dates):
      # Initial possible dates
      possible_dates = dates.copy()
      
      # Step 1: Albert's statement
      # Create a count of each day
      day_counts = {}
      for month, day in dates:
          day_counts[day] = day_counts.get(day, 0) + 1
  
      # Filter out months where a unique day exists (Albert knows Bernard doesn't know)
      possible_months = set()
      for month in set(month for month, day in dates):
          month_days = [day for m, day in dates if m == month]
          if not any(day_counts[day] == 1 for day in month_days):
              possible_months.add(month)

      possible_dates = [ (month, day) for (month, day) in possible_dates if month in possible_months ]
  
      # Step 2: Bernard's statement
      # Recount the days in the filtered possible dates
      day_counts_in_possible = {}
      for month, day in possible_dates:
          day_counts_in_possible[day] = day_counts_in_possible.get(day, 0) + 1

      # Bernard can now deduce the date; keep dates where the day is unique
      possible_dates = [ (month, day) for (month, day) in possible_dates if day_counts_in_possible[day] == 1 ]

      # Step 3: Albert's final statement
      # Recount the months in the possible dates
      month_counts_in_possible = {}
      for month, day in possible_dates:
          month_counts_in_possible[month] = month_counts_in_possible.get(month, 0) + 1

      # Albert now knows the date; keep dates where the month is unique
      possible_dates = [ (month, day) for (month, day) in possible_dates if month_counts_in_possible[month] == 1 ]

      # The remaining date is Cheryl's birthday
      if len(possible_dates) == 1:
          return possible_dates[0]
      else:
          return None
  
  # Solve the problem
  birthday = solve_cheryls_birthday(dates)
  if birthday:
      print(f"Cheryl's birthday is on {birthday[0]} {birthday[1]}")
  else:
      print("Unable to determine Cheryl's birthday.")
  ```

Output:

Cheryl's birthday is on July 16

How does solving a logic puzzle imply a theory of mind? I don’t mean to say that LLMs don’t have a theory of mind, just that deductive reasoning does not amount to empathetic evaluations of how someone else thinks and feels…

…unless you’re a programmer.

Unfortunately they do show up in tech interviews.

“What’s the correct way to delete an element from a Java array while iterating over it?”

“Well I suppose you’d need to avoid invalidating the iterator state but I don’t recall the…”

“BZZT!”

Disappointing that Norvig didn’t use the model that OpenAI states is their best model for programming.

Also using himself as the programmer seemed like a convenient choice. I’d much rather see him grab a random professional programmer for the task.

No it doesn't.

You'd get the answer to a riddle wrong or miss something and nobody would start assuming that means you lack a fundamental understanding of how the world works. There's entire fields that look into how and why we make various mistakes and riddles and puzzles designed specifically to trip people up.

If you want to test if these models can solve riddles, or where they make mistakes go right ahead that's great. It's any assumption it has a much deeper meaning that is wrong to me.