Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Diffusion models are real-time game engines

gamengen.github.io

Post reply on HN
Print view View original

241–250 of 430 posts

Re: Diffusion models are real-time game engines

#241

Post by freestyle24147 »

panki27 wrote: post #120

> Human raters are only slightly better than random chance at distinguishing short clips of the game from clips of the simulation. I can hardly believe this claim, anyone who has played some amount of DOOM before should notice the viewport and textures not "feeling right", or the usually static objects moving slightly.

It made me laugh. Maybe they pulled random people from the hallway who had never seen the original Doom (or any FPS), or maybe only selected people who wore glasses and forgot them at their desk.

Re: Diffusion models are real-time game engines

#243

Post by TeMPOraL »

wavemode wrote: post #190
vessenes wrote: post #2

So, this is surprising. Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected, which would be roughly ‘none’. Google here uses SD 1.4, as the core of the diffusion model, which is a nice reminder that open models are useful to even giant cloud monopolies. The two main things of note I took away from the summary were: 1) they got infinite training data using agents playing doom…

> Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected To temper this a bit, you may want to pay close attention to the demo videos. The player rarely backtracks, and for good reason - the few times the character does turn around and look back at something a second time, it has changed significantly (the most noticeable I think is the room with the grey wall and triangle sign…

So for the next iteration, they should add a minimap overlay (perhaps on a side channel) - it should help the model give more consistent output in any given location. Right now, the game is very much like a lucid dream - the universe makes sense from moment to moment, but without outside reference, everything that falls out of short-term memory (few frames here) gets reimagined.

Re: Diffusion models are real-time game engines

#244

Post by znx_0 »

kqr wrote: post #232
dtagames wrote: post #224

A diffusion model cannot be a game engine because a game engine can be used to create new games and modify the rules of existing games in real time -- even rules which are not visible on-screen. These tools are fascinating but, as with all AI hype, they need a disclaimer: The tool didn't create the game. It simply generated frames and the appearance of play mechanics from a game it sampled (which humans created).

> even rules which are not visible on-screen. If a rule was changed but it's never visible on the screen, did it really change? > It simply generated frames and the appearance of play mechanics from a game it sampled (which humans created). Simply?! I understand it's mechanically trivial but the fact that it's compressed such a rich conditional distribution seems far from simple to me.

> If a rule was changed but it's never visible on the screen, did it really change?

Well for "some" games it does really change

Re: Diffusion models are real-time game engines

#245

Post by beepbooptheory »

whiteboardr wrote: post #217
wavemode wrote: post #190

Earlier quoted context omitted.

> Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected To temper this a bit, you may want to pay close attention to the demo videos. The player rarely backtracks, and for good reason - the few times the character does turn around and look back at something a second time, it has changed significantly (the most noticeable I think is the room with the grey wall and triangle sign…

But does it need to be frame-based? What if you combine this with an engine in parallel that provides all geometry including characters and objects with their respective behavior, recording changes made through interactions the other model generates, talking back to it? A dialogue between two parties with different functionality so to speak. (Non technical person here - just fantasizing)

What would the model provide if not what we see on the screen?

Re: Diffusion models are real-time game engines

#246

Post by alkonaut »

The job of the game engine is also to render the world given only the worlds properties (textures, geometries, physics rules, ...), and not given "training data that had to be supplied from an already written engine".

I'm guessing that the "This door requires a blue key" doesn't mean that the user can run around, the engine dreams up a blue key in some other corner of the map, and the user can then return to the door and the engine now opens the door? THAT would be impressive. It's interesting to think that all that would be required for that task to go from really hard to quite doable, would be that the door requiring the blue key is blue, and the UI showing some icon indicating the user possesses the blue key. Without that, it becomes (old) hidden state.

Re: Diffusion models are real-time game engines

#247

Post by TeMPOraL »

sharpshadow wrote: post #240
dtagames wrote: post #224

A diffusion model cannot be a game engine because a game engine can be used to create new games and modify the rules of existing games in real time -- even rules which are not visible on-screen. These tools are fascinating but, as with all AI hype, they need a disclaimer: The tool didn't create the game. It simply generated frames and the appearance of play mechanics from a game it sampled (which humans created).

So all it did is generate a video of the gameplay which is slightly different from the video it used for training?

No, it implements a 3D FPS that's interactive, and renders each frame based on your input and a lot of memorized gameplay.

Re: Diffusion models are real-time game engines

#249

Post by wantsanagent »

Anyone have reliable numbers on the file sizes here? Doom.exe from my searches was around 715k, and with all assets somewhere around 10MB. It looks like the SD 1.4 files are over 2GB, so it's likely we're looking at a 200-2000x increase in file size depending on if you think of this as an 'engine' or the full game.

Re: Diffusion models are real-time game engines

#250

Post by meheleventyone »

GaggiX wrote: post #189
meheleventyone wrote: post #139

Earlier quoted context omitted.

It's telling IMO that they only want people opinions based on our notoriously faulty memories rather than sitting comparable situations next to one another in the game and simulation then analyzing them. Several things jump out watching the example video.

>rather than sitting comparable situations next to one another in the game and simulation then analyzing them. That's literally how the human rating was setup if you read the paper.

I think you misunderstand me. I don't mean a snap evaluation and deciding between two very-short competing videos which is what the participants were doing. I mean doing an actual analysis of how well the simulation matches the ground truth of the game.

What I'd posit is that it's not actually a very good replication of the game but very good a replicating short clips that almost look like the game and the short time horizons are deliberately chosen because the authors know the model lacks coherence beyond that.

Post reply on HN