There's been a ton of work to generate assets for games using AI: 3d models, textures, code, etc. None of that may even be necessary with a generative game engine like this! If you could scale this up, train on all games in existence, etc. I bet some interesting things would happen
Diffusion models are real-time game engines
11–20 of 430 posts
Re: Diffusion models are real-time game engines
#12There is no text conditioning provided to the SD model because they removed it, but one can imagine a near future where text prompts are enough to create a fun new game!
Yes they had to use RL to learn what DOOM looks like and how it works, but this doesn’t necessarily pose a chicken vs egg problem. In the same way that LLMs can write a novel story, despite only being trained on existing text.
IMO one of the biggest challenges with this approach will be open world games with essentially an infinite number of possible states. The paper mentions that they had trouble getting RL agents to completely explore every nook and corner of DOOM. Factorio or Dwarf Fortress probably won’t be simulated anytime soon…I think.
Re: Diffusion models are real-time game engines
#13So, diffusion models are game engines as long as you already built the game? You need the game to train the model. Chicken. Egg?
here are some ideas: - you could build a non-real-time version of the game engine and use the neural net as a real-time approximation - you could edit videos shot in real life to have huds or whatever and train the neural net to simulate reality rather than doom. (this paper used 900 million frames which i think is about a year of video if it's 30fps, but maybe algorithmic improvements can cut the training requiremen…
That feels like the endgame of video game generation.
You select an art style, a video and the type of game you'd like to play.
The game is then generated in real-time responding to each action with respect to the existing rule engine.
I imagine a game like that could get so convincing in its details and immersiveness that one could forget they're playing a game.
Re: Diffusion models are real-time game engines
#14You know how when you're dreaming and you walk into a room at your house and you're suddenly naked at school?
I'm convinced this is the code that gives Data (ST TNG) his dreaming capabilities.
Re: Diffusion models are real-time game engines
#15[dead]
Re: Diffusion models are real-time game engines
#16Earlier quoted context omitted.
here are some ideas: - you could build a non-real-time version of the game engine and use the neural net as a real-time approximation - you could edit videos shot in real life to have huds or whatever and train the neural net to simulate reality rather than doom. (this paper used 900 million frames which i think is about a year of video if it's 30fps, but maybe algorithmic improvements can cut the training requiremen…
That feels like the endgame of video game generation. You select an art style, a video and the type of game you'd like to play. The game is then generated in real-time responding to each action with respect to the existing rule engine. I imagine a game like that could get so convincing in its details and immersiveness that one could forget they're playing a game.
Holodeck is just around the corner
Re: Diffusion models are real-time game engines
#17The quest to run doom on everything continues. Technically speaking, isn't this the greatest possible anti-Doom, the Doom with the highest possible hardware requirement? I just find it funny that on a linear scale of hardware specification, Doom now finds itself on both ends.
>Technically speaking, isn't this the greatest possible anti-Doom When I read this part I thought you were going to say because you're technically not running Doom at all. That is, instead of running Doom without Doom's original hardware/software environment (by porting it), you're running Doom without Doom itself.
Pierre Menard, Author of Doom.
Re: Diffusion models are real-time game engines
#18Earlier quoted context omitted.
here are some ideas: - you could build a non-real-time version of the game engine and use the neural net as a real-time approximation - you could edit videos shot in real life to have huds or whatever and train the neural net to simulate reality rather than doom. (this paper used 900 million frames which i think is about a year of video if it's 30fps, but maybe algorithmic improvements can cut the training requiremen…
That feels like the endgame of video game generation. You select an art style, a video and the type of game you'd like to play. The game is then generated in real-time responding to each action with respect to the existing rule engine. I imagine a game like that could get so convincing in its details and immersiveness that one could forget they're playing a game.
EXISTENZ IS PAUSED!
Re: Diffusion models are real-time game engines
#19So in the future we can play FPS games given any setting? Pog
Re: Diffusion models are real-time game engines
#20The quest to run doom on everything continues. Technically speaking, isn't this the greatest possible anti-Doom, the Doom with the highest possible hardware requirement? I just find it funny that on a linear scale of hardware specification, Doom now finds itself on both ends.
> Technically speaking, isn't this the greatest possible anti-Doom, the Doom with the highest possible hardware requirement?
Not really? The greatest anti-Doom would be an infinite nest of these types of models predicting models predicting Doom at the very end of the chain.
The next step of anti-Doom would be a model generating the model, generating the Doom output.