Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why are large language models so terrible at video games?

spectrum.ieee.org

Post reply on HN
Print view View original

61–64 of 64 posts

Re: Why are large language models so terrible at video games?

#61

Post by stared »

meffmadd wrote: post #51
stared wrote: post #38

Earlier quoted context omitted.

Nice! I remember "Baba is Eval" ( https://fi-le.net/baba/ ), released 11 months ago, back when Claude Opus 4 was the strongest model. Back then, I was surprised how poor was it even at the first level. I am happy to see an another approach - and indeed, with much stronger results.

Yes that was the post that inspired me to build this. While I did implement a more comprehensive harness with path finding tools etc. the models themselves have improved significantly.

I saw you mentioned that!

Anyway: did you test it with Claude Opus 4.8?

Re: Why are large language models so terrible at video games?

#62

Post by cubefox »

iwhalen wrote: post #55
kqr wrote: post #39

Earlier quoted context omitted.

But LLMs are terrible at text adventures too. See e.g. https://entropicthoughts.com/updated-llm-benchmark and previous articles referenced in there. I have yet to see any sort of harness that lets a frontier LLM interact with a text adventure and make meaningful progress on its own.

To pile on, they're also bad at games that are 2D text based environments. ARC-AGI-3 shows this: https://arcprize.org/arc-agi/3 I've done some work as well on Rogue (sorry for self-promotion): https://iwhalen.github.io/rogue-bench/

There is no "2D text" processing when it comes to LLMs. They process text as ordinary, sequential 1D text only. And humans process "2D text" like any other 2D image. So 2D text isn't really a thing in any case. Saying LLMs are bad at 2D text is like saying that humans are bad at 2D audio.

Re: Why are large language models so terrible at video games?

#63

Post by BergAndCo »

kqr wrote: post #39
ceheaaf wrote: post #19

It feels like they're really focusing on overstating how confusing and weird it is that an LLM can write code but not play games very well, rather than just explaining it. Code is text. LLMs are text input/output machines. Game input/output is not at all text. LLMs can certainly reason about games with a simple/explicit enough domain (try a risk tournament where models can talk to each other between turns!)

But LLMs are terrible at text adventures too. See e.g. https://entropicthoughts.com/updated-llm-benchmark and previous articles referenced in there. I have yet to see any sort of harness that lets a frontier LLM interact with a text adventure and make meaningful progress on its own.

[dead]
Post reply on HN