I found LLMs to be surprisingly good at puzzle games like Baba Is You: https://meffmadd.github.io/samplesurium/posts/baba_is_agent/
Why are large language models so terrible at video games?
31–40 of 64 posts
Re: Why are large language models so terrible at video games?
#32The coding comparison is more interesting to me. Programming has unusually good feedback loops. A test fails, an exception gets thrown, a benchmark regresses. Most games don't give you that kind of signal. I wonder how much of current coding performance depends on that.
Re: Why are large language models so terrible at video games?
#33As others have hinted at LLMs aren't really made in a way that makes them likely to play video games (CS/Halo and such) well. I wonder how they'd fare "against" text based adventures like Zork (which they'll no doubt have ample knowledge about) and newer text based adventure games (which they'll know less about).
Nethack has been widely used to test reinforcement learning agents, starting from at least 2020; there was a Nethack challenge at NeurIPS 2021. https://nethackchallenge.com/report.html
For a more recent test, see https://kenforthewin.github.io/blog/posts/nethack-agent/ .
Re: Why are large language models so terrible at video games?
#34Earlier quoted context omitted.
Yet LLMs can play chess and have a "mental" representation of the chessboard. If LLMs get better but do not progress at playing games when not specifically trained on it it seems to point to a generalisation failure, a limitation that would prevent LLMs to ever achieve AGI, I do not know if that is weird but it seems that for now nobody really knows if they can achieve AGI or not. Perhaps some emergent behavior will…
Chess games are in their training set, other games are not.
Chess is representable entirely in text as well, and generally speaking the LLM concept of "picking the next best token" fits pretty well for "picking the next best move" where a move is a text token
Re: Why are large language models so terrible at video games?
#35Earlier quoted context omitted.
Chess games are in their training set, other games are not.
Chess is representable entirely in text as well, and generally speaking the LLM concept of "picking the next best token" fits pretty well for "picking the next best move" where a move is a text token
That representation is also old, incredibly well documented, and used to describe how to reason about chess. There are of course text guides to other games in training data but they rely upon depictions of what’s happening that aren’t purely text so the game harness is always going to have to make novel decisions about represent the game as text.
Re: Why are large language models so terrible at video games?
#36The coding comparison is more interesting to me. Programming has unusually good feedback loops. A test fails, an exception gets thrown, a benchmark regresses. Most games don't give you that kind of signal. I wonder how much of current coding performance depends on that.
I have been noting this as well. It also had an unfair advantage of having all of open source code to train on, and a bunch of human discussions about code quality and structure. Now as well, the feedback loop of us all using coding agents in real life scenarios.
Not many industries except perhaps writing have had that advantage, in many ways coding is one of the best case scenarios for LLMs.
Re: Why are large language models so terrible at video games?
#37Why is a language model bad at video games? I think the answer is stated in the question itself.
Hilliariously correct!
Re: Why are large language models so terrible at video games?
#38I found LLMs to be surprisingly good at puzzle games like Baba Is You: https://meffmadd.github.io/samplesurium/posts/baba_is_agent/
Nice!
I remember "Baba is Eval" (https://fi-le.net/baba/), released 11 months ago, back when Claude Opus 4 was the strongest model. Back then, I was surprised how poor was it even at the first level.
I am happy to see an another approach - and indeed, with much stronger results.
Re: Why are large language models so terrible at video games?
#39It feels like they're really focusing on overstating how confusing and weird it is that an LLM can write code but not play games very well, rather than just explaining it. Code is text. LLMs are text input/output machines. Game input/output is not at all text. LLMs can certainly reason about games with a simple/explicit enough domain (try a risk tournament where models can talk to each other between turns!)
But LLMs are terrible at text adventures too. See e.g. https://entropicthoughts.com/updated-llm-benchmark and previous articles referenced in there.
I have yet to see any sort of harness that lets a frontier LLM interact with a text adventure and make meaningful progress on its own.
Re: Why are large language models so terrible at video games?
#40As others have hinted at LLMs aren't really made in a way that makes them likely to play video games (CS/Halo and such) well. I wonder how they'd fare "against" text based adventures like Zork (which they'll no doubt have ample knowledge about) and newer text based adventure games (which they'll know less about).
They aren't good at Zork[1] and neither at newer and/or more obscure text adventures[2].
[1]: https://www.lowimpactfruit.com/p/zork-bench-an-llm-reasoning...
[2]: https://entropicthoughts.com/evaluating-llms-playing-text-ad...