Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI's o1 Playing Codenames

suveenellawela.com

Post reply on HN
Print view View original

11–20 of 138 posts

Re: OpenAI's o1 Playing Codenames

#11

Post by Smaug123 »

Klaster_1 wrote: post #8
tsroe wrote: post #6

Fun quirk about this game: If there aren't too many cards left and your teammate knows their powers of two, you have a winning strategy. You simply lay a mental bitmap over all remaining cards, setting 1 for cards that belong to your team and 0 for all others. You can then just say the number that is represented by this bitmap, e.g. "five" for 0101, and your teammate can decode it in their head. All numbers are, afte…

Guys I was playing with declared a similar move against the rules, so it was back to the old latent space search.

It is explicitly against the rules (https://czechgames.com/files/rules/codenames-rules-en.pdf), so they were correct. "Your clue must be about the meaning of the words. You can't use your clue to talk about the letters in a word or its position on the table."

Re: OpenAI's o1 Playing Codenames

#14

Post by pama »

xnickb wrote: post #12

Somehow I expected AI to give clues that combine 4-5-6 words at a time. It's not at all impressive to me. And I'm not a serious player at all

I was wondering about the same. It is possible that the instructions didn’t try to make the gameplay as aggressive as possible. A good model could optimize the separator to make it easy to guess the most words possible. By having access to its own state, it should be possible to reach 5–6 words in most cases. There is an argument for keeping words around that would increase the difficulty of the opponents guessing large/clean separations, so it is possible that optimal play includes simple pairs on occasion. Very interesting application nonetheless.

Re: OpenAI's o1 Playing Codenames

#15

Post by pieix »

suveen_ellawela wrote: post #3
JaggerFoo wrote: post #2

I did this with Claude over the holidays. Putting Claude in the role as a guesser and comparing the guess to another experience human player. It turns out they both matched each other.

That's a nice experiment! I think codenames could definietly be an evaluation method for LLMs.

Elo on different card games/board games would be a great eval metric now that the systems are general enough to play Codenames, chess, poker…

Re: OpenAI's o1 Playing Codenames

#16

Post by fercircularbuf »

I've intuitively felt that this general class of task is what these LLMs are absolutely best at. I'm not an expert on these things, but isn't this thanks to word embeddings and how words are mapped into high dimensional vector space within the model? I would imagine that because every word is mapped this way, finding a word that exists in the same area as mail, lawyer, log, and line in some vector space would be trivial for the model to do, right?

Re: OpenAI's o1 Playing Codenames

#17

Post by deredede »

croes wrote: post #13

Is that really surprising? It’s basically the same brain playing with itself. Seems quite natural to link the code names to the same words. Let different LLMs play.

This is the take I thought I'd have, but in the last example, the guesser model reaches the correct conclusion using a different reasoning than the clue giver model.

The clue giver justifies the link of Paper and Log as "written records", and between Paper and Line as "lines of text". But the guesser model connects Paper and Log because "paper is made from logs" (reaching the conclusion through a different meaning of Log), and connects Paper and Line because "'lined paper' is a common type of paper".

Similarly, in the first example, the clue giver connects Monster and Lion because lions are "often depicted as a mythical beast or monster in legends" (a tenuous connection if you ask me), whereas the guesser model thought about King because of King Kong (which I also prefer to Lion).

Re: OpenAI's o1 Playing Codenames

#18

Post by ushiroda80 »

croes wrote: post #13

Is that really surprising? It’s basically the same brain playing with itself. Seems quite natural to link the code names to the same words. Let different LLMs play.

Yeah not sure what’s impressive about this. Having the model be both the guesser and clue giver will of course have good results as it’s simply a reflections of o1’s weighting of tokens.

Interestingly this could be a way to potentially reverse engineer o1’s weightings

Re: OpenAI's o1 Playing Codenames

#19

Post by andrepd »

tsroe wrote: post #6

Fun quirk about this game: If there aren't too many cards left and your teammate knows their powers of two, you have a winning strategy. You simply lay a mental bitmap over all remaining cards, setting 1 for cards that belong to your team and 0 for all others. You can then just say the number that is represented by this bitmap, e.g. "five" for 0101, and your teammate can decode it in their head. All numbers are, afte…

This is explicitly against the rules.
Post reply on HN