Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Diffusion models are real-time game engines

gamengen.github.io

Post reply on HN
Print view View original

191–200 of 430 posts

Re: Diffusion models are real-time game engines

#191

Post by rvnx »

refibrillator wrote: post #51
vessenes wrote: post #2

So, this is surprising. Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected, which would be roughly ‘none’. Google here uses SD 1.4, as the core of the diffusion model, which is a nice reminder that open models are useful to even giant cloud monopolies. The two main things of note I took away from the summary were: 1) they got infinite training data using agents playing doom…

Just want to clarify a couple possible misconceptions: The diffusion model doesn’t maintain any state itself, though its weights may encode some notion of cause/effect. It just renders one frame at a time (after all it’s a text to image model, not text to video). Instead of text, the previous states and frames are provided as inputs to the model to predict the next frame. Noise is added to the previous frames before…

The concept is that if you train a Diffusion model by feeding all the possible frames seen in the game.

The training was over almost 1 billion frames, 20 days of full-time play-time, taking a screenshot of every single inch of the map.

Now you show him N frames as input, and ask it "give me frame N+1", then it gives you the frame n. N+1 back based on how it was originally seen during training.

But it is not frame N+1 from a mysterious intelligence, it's simply frame N+1 given back from past database.

The drift you mentioned is actually a clear (but sad) proof that the model does not work at inventing new frames, and can only spit out an answer from the past dataset.

It's a bit like if you train stable diffusion on Simpsons episodes, and that it outputs the next frame of an existing episode that was in the training set, but few frames later goes wild and buggy.

Re: Diffusion models are real-time game engines

#192

Post by bergen »

PoignardAzur wrote: post #163
nine_k wrote: post #134

Earlier quoted context omitted.

But it's not a game. It's a memory of a game video, predicting the next frame based on the few previous frames, like "I can imagine what happened next". I would call it the world's least efficient video compression. What I would like to see is the actual predictive strength, aka imagination, which I did not notice mentioned in the abstract. The model is trained on a set of classic maps. What would it do, given a few…

> But it's not a game. It's a memory of a game video, predicting the next frame based on the few previous frames, like "I can imagine what happened next". It's not super clear from the landing page, but I think it's an engine? Like, its input is both previous images and input for the next frame. So as a player, if you press "shoot", the diffusion engine need to output an image where the monster in front of you takes…

How is what you think they say not clear?

We present GameNGen, the first game engine powered entirely by a neural model that enables real-time interaction with a complex environment over long trajectories at high quality.

Re: Diffusion models are real-time game engines

#193

Post by HellDunkel »

MasterScrat wrote: post #136
HellDunkel wrote: post #118

Although impressive i must disagree. Diffusion models are not game engines. A game engine is a component to propell your game (along the time axis?). In that sense it is similar to the engine of the car, hence the name. It does not need a single working car nor a road to drive on do its job. The above is a dynamic, interactive replication of what happens when you put a car on a given road, requiring a million test dr…

Interesting point. In a way this is a "simulated game engine", trained from actual game engine data. But I would argue a working simulated game engine becomes a game engine of its own, as it is then able to "propell the game" as you say. The way it achieves this becomes irrelevant, in one case the content was crafted by humans, in the other case it mimics existing game content, the player really doesn't care! > An en…

The game doom is more than a game engine, isnt it? I‘d be okay with calling the above a „simulated game“ or a „game“. My point is: let‘s not conflate the idea of a „game engine“ which is a construct of intellectual concepts put together to create a simulation of „things happening in time“ and deriving output (audio and visual). the engine is fed with input and data (levels and other assets) and then drives(EDIT) a „game“.

training the model with a final game will never give you an engine. maybe a „simulated game“ or even a „game“ but certainly not an „engine“. the latter would mean the model would be capable to derive and extract the technical and intellectual concepts and apply them elsewhere.

Re: Diffusion models are real-time game engines

#195

Post by kragen »

kqr wrote: post #180
injidup wrote: post #68

Earlier quoted context omitted.

Why games? I will train it on 1 years worth of me attending Microsoft teams meetings. Then I will go surfing.

Even if you spend 40 hours a week in video conferences, you'll have to work for over four years to get one years' worth of footage. Of course, by then the models will be even better and so you might actually have a chance of going surfing. I guess I should start hoarding video of myself now.

the neural net doesn't need a year of video to train to simulate your face; it can do that from a single photo. the year of video is to learn how to play the game, and in most cases lots of people are playing the same game, so you can dump all their video in the same training set

Re: Diffusion models are real-time game engines

#196

Post by lackoftactics »

I think Alan's conservative countdown to AGI will need to be updated after this. https://lifearchitect.ai/agi/ This is really impressive stuff. I thought about it a couple of months ago, that probably this is the next modality worth exploring for data, but didn't imagine it would come so fast. On the other side, the amount of compute required is crazy.

Re: Diffusion models are real-time game engines

#197

Post by jetrink »

rvnx wrote: post #191
refibrillator wrote: post #51

Earlier quoted context omitted.

Just want to clarify a couple possible misconceptions: The diffusion model doesn’t maintain any state itself, though its weights may encode some notion of cause/effect. It just renders one frame at a time (after all it’s a text to image model, not text to video). Instead of text, the previous states and frames are provided as inputs to the model to predict the next frame. Noise is added to the previous frames before…

The concept is that if you train a Diffusion model by feeding all the possible frames seen in the game. The training was over almost 1 billion frames, 20 days of full-time play-time, taking a screenshot of every single inch of the map. Now you show him N frames as input, and ask it "give me frame N+1", then it gives you the frame n. N+1 back based on how it was originally seen during training. But it is not frame N+1…

I don't think you've understood the project completely. The model accepts player input, so frame 601 could be quite different if the player decided to turn left rather than right, or chose that moment to fire at an exploding barrel.

Re: Diffusion models are real-time game engines

#198

Post by hypertele-Xii »

SeanAnderson wrote: post #64
slashdave wrote: post #56

Earlier quoted context omitted.

It is just video. There are no external interactions. Heck, it is far simpler than video, because the point of view and frame is fixed.

I think you're mistaken. The abstract says it's interactive, "We present GameNGen, the first game engine powered entirely by a neural model that enables real-time interaction" Further - "a diffusion model is trained to produce the next frame, conditioned on the sequence of past frames and actions." specifically "and actions" User input is being fed into this system and subsequent frames take that into account. The us…

Then why do monsters become blurry smudgy messes when shot? That looks like a video compression artifact of a neural network attempting to replicate low-structure image (source material contains guts exploding, very un-structured visual).

Re: Diffusion models are real-time game engines

#199

Post by mensetmanusman »

wavemode wrote: post #190
vessenes wrote: post #2

So, this is surprising. Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected, which would be roughly ‘none’. Google here uses SD 1.4, as the core of the diffusion model, which is a nice reminder that open models are useful to even giant cloud monopolies. The two main things of note I took away from the summary were: 1) they got infinite training data using agents playing doom…

> Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected To temper this a bit, you may want to pay close attention to the demo videos. The player rarely backtracks, and for good reason - the few times the character does turn around and look back at something a second time, it has changed significantly (the most noticeable I think is the room with the grey wall and triangle sign…

That is kind of cool though, I would play like being lost in a dream.

If on the backend you could record the level layouts in memory you could have exploration teams that try to find new areas to explore.

Re: Diffusion models are real-time game engines

#200

Post by mensetmanusman »

rvnx wrote: post #191
refibrillator wrote: post #51

Earlier quoted context omitted.

Just want to clarify a couple possible misconceptions: The diffusion model doesn’t maintain any state itself, though its weights may encode some notion of cause/effect. It just renders one frame at a time (after all it’s a text to image model, not text to video). Instead of text, the previous states and frames are provided as inputs to the model to predict the next frame. Noise is added to the previous frames before…

The concept is that if you train a Diffusion model by feeding all the possible frames seen in the game. The training was over almost 1 billion frames, 20 days of full-time play-time, taking a screenshot of every single inch of the map. Now you show him N frames as input, and ask it "give me frame N+1", then it gives you the frame n. N+1 back based on how it was originally seen during training. But it is not frame N+1…

Research is the acquisition of knowledge that may or may not have practical applications.

They succeeded in the research, gained knowledge, and might be able to do something awesome with it.

It’s a success even if they don’t sell anything.

Post reply on HN