Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI's o1 Playing Codenames

suveenellawela.com

Post reply on HN
Print view View original

101–110 of 138 posts

Re: OpenAI's o1 Playing Codenames

#101

Post by n4r9 »

hunter2_ wrote: post #60
mtmickush wrote: post #49

Earlier quoted context omitted.

Other advanced tactics involve giving a broad clue that matches 3-4 of your own and just one other (either your opponents or a civilian). Your team can pick up all the matches across several turns and the one off doesn't hurt as much as the plus four helps

The S-tier tactic: When that high-number clue is cut short by a turn-ending mistake, the guessers tell their clue giver to inflate the number given during the totally unrelated next clue by however many remained from the truncated turn for which they don't need additional information to locate (and therefore it would be wasteful for a future clue to re-group those) so the stated number of that next clue must allow fo…

Yeah, in fact we tend to play without a limit on the number of guesses, just to avoid this sort of loophole. In variants like Codenames Duet I think there's also no limit on the number of guesses.

Another thing the guessers can do if unsure about one of the tiles from the last round, is to tell the clue giver which tile they think it was. The clue giver then tries to give a clue that either tenuously links to it or clearly excludes it. That can give the clue more scope for linking to several other words. It risks giving information to the other team though so is more of an final turn play.

Re: OpenAI's o1 Playing Codenames

#102

Post by suveen_ellawela »

lolinder wrote: post #52

A small weakness in this test is that one of the keys to strategic Codenames play is understanding your partner. You're not just trying to connect the words, you're trying to connect them in a way that will be obvious to your partner. As a computing analogy: you're trying to serialize a few cards in a way that will be deserializable by the other player. This test pairs o1 with itself, which means the serializer is th…

thanks for the comment. I actually tried explicitly mentioning in the prompt that 'Your guesser follows the same reasoning process'. But this did not make any clear improvements. Maybe I should've done more prompt engineering.

Re: OpenAI's o1 Playing Codenames

#103

Post by suveen_ellawela »

unlikelymordant wrote: post #22
deredede wrote: post #17

Earlier quoted context omitted.

This is the take I thought I'd have, but in the last example, the guesser model reaches the correct conclusion using a different reasoning than the clue giver model. The clue giver justifies the link of Paper and Log as "written records", and between Paper and Line as "lines of text". But the guesser model connects Paper and Log because "paper is made from logs" (reaching the conclusion through a different meaning of…

generally there is a "temperature" parameter that can be used to add some randomness or variety to the LLMs outputs by changing the likelihood of the next word being selected. This means you could just keep regenerating the same response and get different answers each time. each time it will give different plausible responses, and this is all from the same model. This doesn't mean it believes any of them, it just kee…

I wanted to play around with the temperature, but unfortunately o1 only supports '1' as the value.

Re: OpenAI's o1 Playing Codenames

#104

Post by suveen_ellawela »

kennyloginz wrote: post #5

Could this just be a case of Reddit being included in the training data? “ I read through codenames official rules to see if using "007" as a clue was allowed, and it turns out it is! To my surprise, I even came across a Reddit post where people were discussing and justifying why this clue fits perfectly within the rules.”

that is a really interesting point. if it is true, this shows direct usage of a single training data point ( cus there are no other resources talking about this fact)

Re: OpenAI's o1 Playing Codenames

#106

Post by suveen_ellawela »

pieix wrote: post #15
suveen_ellawela wrote: post #3

Earlier quoted context omitted.

That's a nice experiment! I think codenames could definietly be an evaluation method for LLMs.

Elo on different card games/board games would be a great eval metric now that the systems are general enough to play Codenames, chess, poker…

totally agree!

Re: OpenAI's o1 Playing Codenames

#107

Post by suveen_ellawela »

__MatrixMan__ wrote: post #39
suveen_ellawela wrote: post #3

Earlier quoted context omitted.

That's a nice experiment! I think codenames could definietly be an evaluation method for LLMs.

It would be fun to build one, perhaps mediated by an app, where you have to guess whether your spymaster is a human or an AI based on the quality of their choices.

i thought of adding a feature where you can get your own spy master. you can give it all your personal info and the clues would be customized. the botteleneck is the other human spymaster has to help with updating the game state cus I(guesser) can't look at the spy master view.
Post reply on HN