Diffusion models are real-time game engines
gamengen.github.io
Diffusion models are real-time game engines
1–10 of 430 posts
Re: Diffusion models are real-time game engines
#2So, this is surprising. Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected, which would be roughly ‘none’. Google here uses SD 1.4, as the core of the diffusion model, which is a nice reminder that open models are useful to even giant cloud monopolies.
The two main things of note I took away from the summary were: 1) they got infinite training data using agents playing doom (makes sense), and 2) they added Gaussian noise to source frames and rewarded the agent for ‘correcting’ sequential frames back, and said this was critical to get long range stable ‘rendering’ out of the model.
That last is intriguing — they explain the intuition as teaching the model to do error correction / guide it to be stable.
Finally, I wonder if this model would be easy to fine tune for ‘photo realistic’ / ray traced restyling — I’d be super curious to see how hard it would be to get a ‘nicer’ rendering out of this model, treating it as a doom foundation model of sorts.
Anyway, a fun idea that worked! Love those.
Re: Diffusion models are real-time game engines
#3The quest to run doom on everything continues.
Technically speaking, isn't this the greatest possible anti-Doom, the Doom with the highest possible hardware requirement?
I just find it funny that on a linear scale of hardware specification, Doom now finds itself on both ends.
Re: Diffusion models are real-time game engines
#4So, diffusion models are game engines as long as you already built the game? You need the game to train the model. Chicken. Egg?
Re: Diffusion models are real-time game engines
#5So, diffusion models are game engines as long as you already built the game? You need the game to train the model. Chicken. Egg?
maybe the next step is adding text guidance and generating non-existing games.
Re: Diffusion models are real-time game engines
#6[deleted]
Re: Diffusion models are real-time game engines
#7There is going to be a flood of these dreamlike "games" in the next few years. This feels likes a bit of a breakthrough in the engineering of these systems.
Re: Diffusion models are real-time game engines
#8The quest to run doom on everything continues. Technically speaking, isn't this the greatest possible anti-Doom, the Doom with the highest possible hardware requirement? I just find it funny that on a linear scale of hardware specification, Doom now finds itself on both ends.
>Technically speaking, isn't this the greatest possible anti-Doom
When I read this part I thought you were going to say because you're technically not running Doom at all. That is, instead of running Doom without Doom's original hardware/software environment (by porting it), you're running Doom without Doom itself.
Re: Diffusion models are real-time game engines
#9It's insane that that this works, and that it works fast enough to render at 20 fps. It seems like they almost made a cross between a diffusion model and an RNN, since they had to encode the previous frames and actions and feed it into the model at each step.
Abstractly, it's like the model is dreaming of a game that it played a lot of, and real time inputs just change the state of the dream. It makes me wonder if humans are just next moment prediction machines, with just a little bit more memory built in.
Re: Diffusion models are real-time game engines
#10So, diffusion models are game engines as long as you already built the game? You need the game to train the model. Chicken. Egg?
here are some ideas:
- you could build a non-real-time version of the game engine and use the neural net as a real-time approximation
- you could edit videos shot in real life to have huds or whatever and train the neural net to simulate reality rather than doom. (this paper used 900 million frames which i think is about a year of video if it's 30fps, but maybe algorithmic improvements can cut the training requirements down) and a year of video isn't actually all that much—like, maybe you could recruit 500 people to play paintball while wearing gopro cameras with accelerometers and gyros on their heads and paintball guns, so that you could get a year of video in a weekend?