Uhhh… demos would be more convincing with enemies and decreasing health
Diffusion models are real-time game engines
71–80 of 430 posts
Re: Diffusion models are real-time game engines
#72So, diffusion models are game engines as long as you already built the game? You need the game to train the model. Chicken. Egg?
here are some ideas: - you could build a non-real-time version of the game engine and use the neural net as a real-time approximation - you could edit videos shot in real life to have huds or whatever and train the neural net to simulate reality rather than doom. (this paper used 900 million frames which i think is about a year of video if it's 30fps, but maybe algorithmic improvements can cut the training requiremen…
The Cloud Gaming platforms could record things for training data.
Re: Diffusion models are real-time game engines
#73It's insane that that this works, and that it works fast enough to render at 20 fps. It seems like they almost made a cross between a diffusion model and an RNN, since they had to encode the previous frames and actions and feed it into the model at each step. Abstractly, it's like the model is dreaming of a game that it played a lot of, and real time inputs just change the state of the dream. It makes me wonder if hu…
Did they take in the entire history as context?
Re: Diffusion models are real-time game engines
#74Earlier quoted context omitted.
>Technically speaking, isn't this the greatest possible anti-Doom When I read this part I thought you were going to say because you're technically not running Doom at all. That is, instead of running Doom without Doom's original hardware/software environment (by porting it), you're running Doom without Doom itself.
Pierre Menard, Author of Doom.
that took a moment, thank you
Re: Diffusion models are real-time game engines
#75An implementation of the game engine in the model itself is theoretically the most accurate solution for predicting the next frame. I'm wondering when people will apply this to other areas like the real world. Would it learn the game engine of the universe (ie physics)?
There has definitely been research for simulating physics based on observation, especially in fluid dynamics but also for rigid body motion and collision. It's important for robotics applications actually. You can bet people will be applying this technique in those contexts.
I think for real world application one challenge is going to be the "action" signal which is a necessary component of the conditioning signal that makes the simulation reactive. In video games you can just record the buttons, but for real world scenarios you need difficult and intrusive sensor setups for recording force signals.
(Again for robotics though maybe it's enough to record the motor commands, just that you can't easily record the "motor commands" for humans, for example)
Re: Diffusion models are real-time game engines
#76Earlier quoted context omitted.
With enough computation, your neural net weights would converge to some very compressed latent representation of the source code of DOOM. Maybe smaller even than the source code itself? Someone in the field could probably correct me on that. At which point, you effectively would be interpolating in latent space through the source code to actually "render" the game. You'd have an entire latent space computer, with an…
The source code lacks information required to render the game. Textures for example.
Obviously assets would get encoded too, in some form. Not necessarily corresponding to the original bitmaps, if the game does some consistent post-processing, the encoded thing would more likely be (equivalent to) the post-processed state.
Re: Diffusion models are real-time game engines
#77There is no text conditioning provided to the SD model because they removed it, but one can imagine a near future where text prompts are enough to create a fun new game! Yes they had to use RL to learn what DOOM looks like and how it works, but this doesn’t necessarily pose a chicken vs egg problem. In the same way that LLMs can write a novel story, despite only being trained on existing text. IMO one of the biggest…
Most games are conditioned on text, it's just that we call it "source code" :).
(Jk of course I know what you mean, but you can seriously see text prompts as compressed forms of programming that leverage the model's prior knowledge)
Re: Diffusion models are real-time game engines
#78The quest to run doom on everything continues. Technically speaking, isn't this the greatest possible anti-Doom, the Doom with the highest possible hardware requirement? I just find it funny that on a linear scale of hardware specification, Doom now finds itself on both ends.
> the Doom with the highest possible hardware requirement?
Isn't that possible by setting arbitrarily high goals for ray-cast rendering?
Re: Diffusion models are real-time game engines
#79There's been a ton of work to generate assets for games using AI: 3d models, textures, code, etc. None of that may even be necessary with a generative game engine like this! If you could scale this up, train on all games in existence, etc. I bet some interesting things would happen
But can you grab what this Ai has learned and generate the 3d models, maps and code to turn it into an actual game that can run on a user's PC? That would be amazing.
What would be the point? This model has been trained on an existing game, so turning it back into assets, maps, and code would just give you a copy of the original game you started with. I suppose you could create variations of it then... but:
You don't even need to do all of that - this trained model already is the game, i.e., it's interactive, you can play the game.
Re: Diffusion models are real-time game engines
#80So, any given sequence of inputs is rebuilt into a corresponding image, twenty times per second. I wonder how separate the game logic and the generated graphics are in the fully trained model. Given a sufficient enough separation between these two, couldn't you basically boil the game/input logic down to an abstract game template? Meaning, you could just output a hash that corresponds to a specific combination of inp…
I think you've just encoded the title of the paper