41–50 of 138 posts
Earlier quoted context omitted.
That's a nice experiment! I think codenames could definietly be an evaluation method for LLMs.
It would be fun to build one, perhaps mediated by an app, where you have to guess whether your spymaster is a human or an AI based on the quality of their choices.
Now obviously it’s still pretty decent at finding the clues. Probably better than a random human who hasn’t played much. Just I find the post’s level of hype overstated. It feels like the author isn’t very experienced with Codenames.
It would be interesting to compare AI:human vs human:human games to see which does better. It seems like AI:AI will overstate its success.
Is that really surprising? It’s basically the same brain playing with itself. Seems quite natural to link the code names to the same words. Let different LLMs play.
I don’t find this “super good”. It’s mostly giving 2 clues which is the most basic level of competence. The paper 4 clue is reasonable but a bit lucky (eg Jack is also a good guess). I also don’t see it actually using tactics properly, which I would consider part of being “super good”. The game isn’t just about picking a good clue each round! Now obviously it’s still pretty decent at finding the clues. Probably bette…
When I play, it's mostly about getting a good 2 clue each time. Then if you can opportunistically get a 3 or 4, that's awesome.
Some tactics come in for choosing the right pairs of 2's so you don't end up mismatched, or leaving clues that might be ambiguous with your opponent's... But that's mostly it.
It'll be fun for multiplayer! Just like how in other online games you can add in a AI to play as one of the players.
Earlier quoted context omitted.
It would be fun to build one, perhaps mediated by an app, where you have to guess whether your spymaster is a human or an AI based on the quality of their choices.
The average human is quite bad. It really works well when the spymaster is (a) experienced and (b) familiar with the other players.
Somehow I expected AI to give clues that combine 4-5-6 words at a time. It's not at all impressive to me. And I'm not a serious player at all
I am similarly less-than-impressed. If you click through to the website, you can watch the replay of one of the games mentioned in the article (the one with the clue "invader"). In that instance, the clues all matched 2-3 words, and the winning team got lucky twice (they guessed an unclued word using an unintended correlation, and their opponent guessed a different one of their unclued words.) You also see a number o…
I don’t find this “super good”. It’s mostly giving 2 clues which is the most basic level of competence. The paper 4 clue is reasonable but a bit lucky (eg Jack is also a good guess). I also don’t see it actually using tactics properly, which I would consider part of being “super good”. The game isn’t just about picking a good clue each round! Now obviously it’s still pretty decent at finding the clues. Probably bette…
Can you elaborate on some of the more advanced tactics? When I play, it's mostly about getting a good 2 clue each time. Then if you can opportunistically get a 3 or 4, that's awesome. Some tactics come in for choosing the right pairs of 2's so you don't end up mismatched, or leaving clues that might be ambiguous with your opponent's... But that's mostly it. It'll be fun for multiplayer! Just like how in other online…
I've been trying to do this with just word2vec, instead of throwing an LLM, since you just need to find a word with the appropriate distances optimized. https://github.com/captn3m0/ideas?tab=readme-ov-file#codenam...
None of them were better than a human at giving hints for 3+ words though