Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why are large language models so terrible at video games?

spectrum.ieee.org

Post reply on HN
Print view View original

51–60 of 64 posts

Re: Why are large language models so terrible at video games?

#51

Post by meffmadd »

stared wrote: post #38
meffmadd wrote: post #31

I found LLMs to be surprisingly good at puzzle games like Baba Is You: https://meffmadd.github.io/samplesurium/posts/baba_is_agent/

Nice! I remember "Baba is Eval" ( https://fi-le.net/baba/ ), released 11 months ago, back when Claude Opus 4 was the strongest model. Back then, I was surprised how poor was it even at the first level. I am happy to see an another approach - and indeed, with much stronger results.

Yes that was the post that inspired me to build this.

While I did implement a more comprehensive harness with path finding tools etc. the models themselves have improved significantly.

Re: Why are large language models so terrible at video games?

#53

Post by meffmadd »

IsTom wrote: post #52
meffmadd wrote: post #31

I found LLMs to be surprisingly good at puzzle games like Baba Is You: https://meffmadd.github.io/samplesurium/posts/baba_is_agent/

I think what should be kept in mind is that these are not the hard levels BIY is famous for.

That is very true but I was surprised by how clear the “signal” was. Only Gemini really confidently solved all levels. But yeah the goal is now to include harder levels as well!

Re: Why are large language models so terrible at video games?

#54

Post by aykutseker »

suyavuz wrote: post #32

The coding comparison is more interesting to me. Programming has unusually good feedback loops. A test fails, an exception gets thrown, a benchmark regresses. Most games don't give you that kind of signal. I wonder how much of current coding performance depends on that.

A failed test often points at the mistake. Most games just tell you that the outcome was bad.

Re: Why are large language models so terrible at video games?

#55

Post by iwhalen »

kqr wrote: post #39
ceheaaf wrote: post #19

It feels like they're really focusing on overstating how confusing and weird it is that an LLM can write code but not play games very well, rather than just explaining it. Code is text. LLMs are text input/output machines. Game input/output is not at all text. LLMs can certainly reason about games with a simple/explicit enough domain (try a risk tournament where models can talk to each other between turns!)

But LLMs are terrible at text adventures too. See e.g. https://entropicthoughts.com/updated-llm-benchmark and previous articles referenced in there. I have yet to see any sort of harness that lets a frontier LLM interact with a text adventure and make meaningful progress on its own.

To pile on, they're also bad at games that are 2D text based environments.

ARC-AGI-3 shows this: https://arcprize.org/arc-agi/3

I've done some work as well on Rogue (sorry for self-promotion): https://iwhalen.github.io/rogue-bench/

Re: Why are large language models so terrible at video games?

#56

Post by duckmysick »

meffmadd wrote: post #31

I found LLMs to be surprisingly good at puzzle games like Baba Is You: https://meffmadd.github.io/samplesurium/posts/baba_is_agent/

Does this use levels from the original game or some custom ones? The solutions to the original levels should be in the training data, be it blogs, reddit comments, or wikis.

Unless the goal was to test how well do the large language models translate solutions in prose to actionable keyboard inputs, which is pretty interesting in itself.

Re: Why are large language models so terrible at video games?

#58

Post by slumberlust »

deyiao wrote: post #28
kingstnap wrote: post #24

Earlier quoted context omitted.

OpenAI's Dota 2 adventures were super hype back in the days.

OpenAI Five doesn’t really know how to play games in general — it only knows how to play Dota.

The only game that matters.

Re: Why are large language models so terrible at video games?

#59

Post by vunderba »

duckmysick wrote: post #56
meffmadd wrote: post #31

I found LLMs to be surprisingly good at puzzle games like Baba Is You: https://meffmadd.github.io/samplesurium/posts/baba_is_agent/

Does this use levels from the original game or some custom ones? The solutions to the original levels should be in the training data, be it blogs, reddit comments, or wikis. Unless the goal was to test how well do the large language models translate solutions in prose to actionable keyboard inputs, which is pretty interesting in itself.

Agreed. I’m surprised how often people seem to miss this. They don’t realize just how gargantuan the training datasets are for these large language models, especially for a very popular game like Baba Is You. I’m sure that both GameFAQs and the Steam forums are in the training data for any reasonably SOTA LLM both of which almost assuredly have complete walkthroughs for BIY.

Re: Why are large language models so terrible at video games?

#60

Post by jiehong »

pixel_popping wrote: post #7
jiehong wrote: post #3

Video games are made to entertain humans, so does it really matter whether LLMs are good at playing them?

It matters a lot because it's a real solution for external bots that plays more "fairly" especially in older games. It also allows to test games autonomously, which is huge if we are talking about automated programming. Imagine if you can bring those AI players to CS 1.6.

Fair point, thanks for changing my point of view :)
Post reply on HN