Its almost like the Large Language Model has trouble with things that arent Language, such as realtime controller input and video output from a game
[dead]
Why are large language models so terrible at video games?
11–20 of 64 posts
Re: Why are large language models so terrible at video games?
#12> This brings us to what seems like a contradiction. LLMs are bad at playing games. Yet at the same time, they’re improving rapidly at coding, a skill set that can be used to create a game. How do these facts fit together? > Togelius: It’s super weird. ...No, it's really not. They're language models. Code is a language. "Playing a game well" is not. One can, hypothetically, encode game inputs in such a way that it se…
Yea it’s wild watching so many smart people convince themselves that LLMs are general purpose AIs. Don’t get me wrong they are incredibly powerful tools. However being surprised that text models cannot play video games particularly well is like being surprised weather models cannot.
Re: Why are large language models so terrible at video games?
#13I wonder if they would be good at text-based games.
Re: Why are large language models so terrible at video games?
#14Earlier quoted context omitted.
It matters a lot because it's a real solution for external bots that plays more "fairly" especially in older games. It also allows to test games autonomously, which is huge if we are talking about automated programming. Imagine if you can bring those AI players to CS 1.6.
LLMs are the wrong tool for video games. There have been plenty of successful non-LLM AIs that have been trained with reinforcement learning to play games. If you want to implement actual bots inside the game, then you want to use explicit logic instead of inferred logic. It's much more efficient and easier to debug. If you want to create Bots for an existing game, which doesn't have its own pre-programmed bots, then…
[dead]
Re: Why are large language models so terrible at video games?
#15I wonder if you paired a few different types of AI together, an LLM agent might be good at strategizing -. E.g. building a strategy on how to handle a scenario. But, it would need to know the entire game manual basically. Then it would pass the stratrgy to a better AI in some way. But it might not be needed if the better gaming AI can just do that part too already.
I admit I know nothing about this though.
Re: Why are large language models so terrible at video games?
#16> This brings us to what seems like a contradiction. LLMs are bad at playing games. Yet at the same time, they’re improving rapidly at coding, a skill set that can be used to create a game. How do these facts fit together? > Togelius: It’s super weird. ...No, it's really not. They're language models. Code is a language. "Playing a game well" is not. One can, hypothetically, encode game inputs in such a way that it se…
Yet LLMs can play chess and have a "mental" representation of the chessboard.
If LLMs get better but do not progress at playing games when not specifically trained on it it seems to point to a generalisation failure, a limitation that would prevent LLMs to ever achieve AGI, I do not know if that is weird but it seems that for now nobody really knows if they can achieve AGI or not. Perhaps some emergent behavior will arise after more scaling.
To me it's only totally unsurprising if you are 100% certain that LLMs will never reach AGI (like LeCun thinks for example).
Re: Why are large language models so terrible at video games?
#17Its almost like the Large Language Model has trouble with things that arent Language, such as realtime controller input and video output from a game
I know someone who tried the "aibot plays pokemon" thing...
From what I saw, even if you frame advance every single frame, they still don't seem to grasp the concept of "I need to hold down this button for a few frames until x happens"...
There's no concept of time, just a never ending state machine thats constantly changing state.
Re: Why are large language models so terrible at video games?
#18Video games are made to entertain humans, so does it really matter whether LLMs are good at playing them?
It matters a lot because it's a real solution for external bots that plays more "fairly" especially in older games. It also allows to test games autonomously, which is huge if we are talking about automated programming. Imagine if you can bring those AI players to CS 1.6.
The headshot/spin bots didn't need ai, all they had to do was ask the server where you were standing, and teleported to your location.
Re: Why are large language models so terrible at video games?
#19It feels like they're really focusing on overstating how confusing and weird it is that an LLM can write code but not play games very well, rather than just explaining it.
Code is text. LLMs are text input/output machines.
Game input/output is not at all text.
LLMs can certainly reason about games with a simple/explicit enough domain (try a risk tournament where models can talk to each other between turns!)
Re: Why are large language models so terrible at video games?
#20Why is a language model bad at video games? I think the answer is stated in the question itself.