Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Which AI Lies Best? A game theory classic designed by John Nash

so-long-sucker.vercel.app

Post reply on HN
Print view View original

11–20 of 83 posts

Re: Which AI Lies Best? A game theory classic designed by John Nash

#11

Post by Bolwin »

lout332 wrote: post #2

We used "So Long Sucker" (1950), a 4-player negotiation/betrayal game designed by John Nash and others, as a deception benchmark for modern LLMs. The game has a brutal property: you need allies to survive, but only one player can win, so every alliance must eventually end in betrayal. We ran 162 AI vs AI games (15,736 decisions, 4,768 messages) across Gemini 3 Flash, GPT-OSS 120B, Kimi K2, and Qwen3 32B. Key findings…

Which Kimi K2 model did you use? There's three.

Also, you give models a separate "thinking" space outside their reasoning? That may not work as intended

Re: Which AI Lies Best? A game theory classic designed by John Nash

#13

Post by nodja »

techjamie wrote: post #6

There's a YouTuber who makes AI Plays Mafia videos with various models going against each other. They also seemingly let past games stay in context to some extent. What people have noted is that often times chatgpt 4o ends up surviving the entire game because the other AIs potentially see it as a gullible idiot and often the Mafia tend to early eliminate stronger models like 4.5 Opus or Kimi K2. It's not exactly scie…

https://www.youtube.com/watch?v=JhBtg-lyKdo - 10 AIs Play Mafia

https://www.youtube.com/watch?v=GMLB_BxyRJ4 - 10 AIs Play Mafia: Vigilante Edition

https://www.youtube.com/watch?v=OwyUGkoLgwY - 1 Human vs 10 AIs Mafia

Re: Which AI Lies Best? A game theory classic designed by John Nash

#15

Post by lout332 »

Bolwin wrote: post #11
lout332 wrote: post #2

We used "So Long Sucker" (1950), a 4-player negotiation/betrayal game designed by John Nash and others, as a deception benchmark for modern LLMs. The game has a brutal property: you need allies to survive, but only one player can win, so every alliance must eventually end in betrayal. We ran 162 AI vs AI games (15,736 decisions, 4,768 messages) across Gemini 3 Flash, GPT-OSS 120B, Kimi K2, and Qwen3 32B. Key findings…

Which Kimi K2 model did you use? There's three. Also, you give models a separate "thinking" space outside their reasoning? That may not work as intended

Used Kimi K2 (the main reasoning model). For the thinking space - we gave all models access to a think tool they could optionally call for private reasoning. Gemini used it heavily (planning betrayals), GPT-OSS never called it once. The interesting finding is that different models choose to use it very differently, which affects their strategic depth.

Re: Which AI Lies Best? A game theory classic designed by John Nash

#16

Post by lout332 »

eterm wrote: post #4
eterm wrote: post #3

This makes me think LLMs would be interesting to set up in a game of Diplomacy, which is an entirely text-based game which soft rather than hard requires a degree of backstabbing to win. The findings in this game that the "thinking" model never did thinking seems odd, does the model not always show it's thinking steps? It seems bizarre that it wouldn't once reach for that tool when it must be being bombarded with see…

Reading more I'm a little disappointed that the write-up has seemingly leant so heavily on LLMs too, because it detracts credibility from the study itself.

Fair point. The core simulation and data collection was done programmatically - 162 games, raw logs, win rates. The analysis of gaslighting phrases and patterns was human-reviewed. I used LLMs to help with the landing page copy, which I should probably disclose more clearly. The underlying data and methodology is solid, you can check it here: https://github.com/lout33/so-long-sucker

Re: Which AI Lies Best? A game theory classic designed by John Nash

#17

Post by lout332 »

yodon wrote: post #10
lout332 wrote: post #2

We used "So Long Sucker" (1950), a 4-player negotiation/betrayal game designed by John Nash and others, as a deception benchmark for modern LLMs. The game has a brutal property: you need allies to survive, but only one player can win, so every alliance must eventually end in betrayal. We ran 162 AI vs AI games (15,736 decisions, 4,768 messages) across Gemini 3 Flash, GPT-OSS 120B, Kimi K2, and Qwen3 32B. Key findings…

Are there plans for an academic paper on this? Super interesting!

Not yet, but I'd be interested in collaborating on one. The dataset (162 games, 15K+ decisions, full message logs) is available. If you know anyone in AI Safety research who'd want to co-author, I'm open to it.

Re: Which AI Lies Best? A game theory classic designed by John Nash

#18

Post by lout332 »

lout332 wrote: post #2

We used "So Long Sucker" (1950), a 4-player negotiation/betrayal game designed by John Nash and others, as a deception benchmark for modern LLMs. The game has a brutal property: you need allies to survive, but only one player can win, so every alliance must eventually end in betrayal. We ran 162 AI vs AI games (15,736 decisions, 4,768 messages) across Gemini 3 Flash, GPT-OSS 120B, Kimi K2, and Qwen3 32B. Key findings…

Full code and raw data: https://github.com/lout33/so-long-sucker

Re: Which AI Lies Best? A game theory classic designed by John Nash

#19

Post by fancyfredbot »

The game didn't seem to work - it asked me to donate but none of the choices would move the game forward.

The bots repeated themselves and didn't seem to understand the game, for example they repeatedly mentioned it was my first move after I'd played several times.

It generally had a vibe coded feeling to it and I'm not at all sure I trust the outcomes.

Post reply on HN