I imagine this is because Tetris is visual and the Gemini models are strong visually.
I figure OP would try and give the models pure text forms of the game?
.....
l....
l....
l.ttt
l..t.
Show HN: TetrisBench – Gemini Flash reaches 66% win rate on Tetris against Opus
11–20 of 42 posts
Re: Show HN: TetrisBench – Gemini Flash reaches 66% win rate on Tetris against Opus
#12Interesting but frustratingly vague on details. How exactly are the models playing? Is it using some kind of PGN equivalent in Tetris that represents a on-going game, passing an ASCII representation, encoding as a JSON structure, or just directly sending screenshots of the game to the various LLMs?
Re: Show HN: TetrisBench – Gemini Flash reaches 66% win rate on Tetris against Opus
#13It would be more interesting to make it build a chess engine and compare it against Stockfish. The chess engine should be a standalone no-dependencies C/C++ program that fits in NNN lines of code.
Comparing against stockfish isn't fair. That's comparing against enormous amounts of compute spent experimenting with strategies, training neutral nets, etc.
It will lose so badly there will be no point in the comparison.
Besides you could compare models (and harnesses) directly against eachother.
Re: Show HN: TetrisBench – Gemini Flash reaches 66% win rate on Tetris against Opus
#14There are some concepts clashing here. I mean, if you let the LLM build a testris bot, it would be 1000x better than what the LLMs are doing. So yes, it is fun to win against an AI, but to be fair against such processing power, you should not be able to win. It is only possible because LLMs are not built for such tasks.
Fun fact: Humans were not build for playing Tetris either!
Re: Show HN: TetrisBench – Gemini Flash reaches 66% win rate on Tetris against Opus
#15I'd like to see a nethackbench.
Re: Show HN: TetrisBench – Gemini Flash reaches 66% win rate on Tetris against Opus
#16Interesting but frustratingly vague on details. How exactly are the models playing? Is it using some kind of PGN equivalent in Tetris that represents a on-going game, passing an ASCII representation, encoding as a JSON structure, or just directly sending screenshots of the game to the various LLMs?
It has to be turn-based. Even with Flash's speed, the inference latency would kill you in a real-time loop. They're likely pausing the game state after every tick to wait for the API response before resuming.
Re: Show HN: TetrisBench – Gemini Flash reaches 66% win rate on Tetris against Opus
#17Thanks for all the questions! More details on how this works:
- Each model starts with an initial optimization function for evaluating Tetris moves.
- As the game progresses, the model sees the current board state and updates its algorithm—adapting its strategy based on how the game is evolving.
- The model continuously refines its optimizer. It decides when it needs to re-evaluate and when it should implement the next optimization function
- The model generates updated code, executes it to score all placements, and picks the best move.
- The reason I reframed this problem to a coding problem is Tetris is an optimization game in nature. At first I did try asking LLMs where to place each piece at every turn but models are just terrible at visual reasoning. What LLMs great at though is coding.
Re: Show HN: TetrisBench – Gemini Flash reaches 66% win rate on Tetris against Opus
#18It would be more interesting to make it build a chess engine and compare it against Stockfish. The chess engine should be a standalone no-dependencies C/C++ program that fits in NNN lines of code.
My back-of-the-envelope guess would be that 99% of LLMs given the task to build a chess engine would probably just end up implementing a flavor of negamax and calling it a day.
Re: Show HN: TetrisBench – Gemini Flash reaches 66% win rate on Tetris against Opus
#19There are some concepts clashing here. I mean, if you let the LLM build a testris bot, it would be 1000x better than what the LLMs are doing. So yes, it is fun to win against an AI, but to be fair against such processing power, you should not be able to win. It is only possible because LLMs are not built for such tasks.
Task: play tetris
Task: write and optimize a tetris bot
Task: write and safely online optimize a tetris bot with consideration for cost to converge
openai/baselines (7 years ago) was leading on RL and then AlphaZero and Self-Attention Transformer networks.
LLMs are trained with RL, but aren't general purpose game theoretic RL agents?
Re: Show HN: TetrisBench – Gemini Flash reaches 66% win rate on Tetris against Opus
#20Interesting but frustratingly vague on details. How exactly are the models playing? Is it using some kind of PGN equivalent in Tetris that represents a on-going game, passing an ASCII representation, encoding as a JSON structure, or just directly sending screenshots of the game to the various LLMs?
answered this in a comment above! It's not turn or visual layout based since LLMs are not trained that way. The representation is a JSON structure, but LLMs plug in algorithms and keeps optimizing it as the game state evolves