Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Diffusion models are real-time game engines

gamengen.github.io

Post reply on HN
Print view View original

251–260 of 430 posts

Re: Diffusion models are real-time game engines

#251

Post by dr_dshiv »

wavemode wrote: post #234
nmstoker wrote: post #223

Earlier quoted context omitted.

I saw a longer video of this that Ethan Mollick posted and in that one, the sequences are longer and they do appear to demonstrate a fair amount of consistency. The clips don't backtrack in the summary video on the paper's home page because they're showing a number of district environments but you only get a few seconds of each. If I studied the longer one more closely, I'm sure inconsistencies would be seen but it s…

Yes it definitely is very good for simulating gameplay footage, don't get me wrong. Its input for predicting the next frame is not just the previous frame, it has access to a whole sequence of prior frames. But to say the model is simulating actual gameplay (i.e. that a person could actually play Doom in this) is far fetched. It's definitely great that the model was able to remember that the gray wall was still there…

It's an empirical question, right? But they didn't do it...

Re: Diffusion models are real-time game engines

#252

Post by commodoreboxer »

rvnx wrote: post #225
OskarS wrote: post #218

Earlier quoted context omitted.

> 1 billion frames in memory... With such dataset, you have seen practically all realistic possibilities in the short-term. I mean... no? Not even close? Multiply the number of game states with the number of inputs at any given frame gives you a number vastly bigger than 1 billion, not even comparable. Even with 20 days of play time to train no, it's entirely likely that at no point did someone stop at a certain loca…

This was the basis for the reasoning: The map 1 has 2'518 walkable map units. There are 65536 angles. 2'518*65'536=165'019'648 If you capture 165M frames, you already cover all the possibilities in terms of camera / player view, but probably the diffusion models don't even need to have all the frames (the same way that LLMs don't).

There's also enemy motion, enemy attacks, shooting, and UI considerations, which make the combinatorials explode.

And Doom movement isn't tile based. The map may be, but you can be in many many places on a tile.

Re: Diffusion models are real-time game engines

#255

Post by alickz »

noiv wrote: post #228
alickz wrote: post #222

Earlier quoted context omitted.

is that something that can be solved with more memory/attention/context? or do we believe it's an inherent limitation in the approach?

I think the real question is does the player get shot from behind?

great question

tangentially related but Grand Theft Auto speedrunners often point the camera behind them while driving so cars don't spawn "behind" them (aka in front of the car)

Re: Diffusion models are real-time game engines

#256

Post by [deleted] »

dtagames wrote: post #224

A diffusion model cannot be a game engine because a game engine can be used to create new games and modify the rules of existing games in real time -- even rules which are not visible on-screen. These tools are fascinating but, as with all AI hype, they need a disclaimer: The tool didn't create the game. It simply generated frames and the appearance of play mechanics from a game it sampled (which humans created).

[deleted]

Re: Diffusion models are real-time game engines

#257

Post by TeMPOraL »

rvnx wrote: post #208
jetrink wrote: post #197

Earlier quoted context omitted.

I don't think you've understood the project completely. The model accepts player input, so frame 601 could be quite different if the player decided to turn left rather than right, or chose that moment to fire at an exploding barrel.

1 billion frames in memory... With such dataset, you have seen practically all realistic possibilities in the short-term. If it would be able to invent action and maps and let the user play "infinite doom", then it would be very different (and impressive!).

Like many people in case of LLMs, you're just demonstrating unawareness of - or disbelief in - the fact that the model doesn't record training data vetbatim, but smears it out in high-dimensional space, from which it then samples. The model then doesn't recall past inputs (which are effectively under extreme lossy compression), but samples from that high-dimensional space to produce output. The high-dimensional representation by necessity captures semantic understanding of the training data.

Generating "infinite Doom" is exactly what this model is doing, as it does not capture the larger map layout well enough to stay consistent with it.

Re: Diffusion models are real-time game engines

#258

Post by jl6 »

bugglebeetle wrote: post #17
fngjdflmdflg wrote: post #8

Earlier quoted context omitted.

>Technically speaking, isn't this the greatest possible anti-Doom When I read this part I thought you were going to say because you're technically not running Doom at all. That is, instead of running Doom without Doom's original hardware/software environment (by porting it), you're running Doom without Doom itself.

Pierre Menard, Author of Doom.

Knee Deep in the Death of the Author.

Re: Diffusion models are real-time game engines

#259

Post by sharpshadow »

TeMPOraL wrote: post #247
sharpshadow wrote: post #240

Earlier quoted context omitted.

So all it did is generate a video of the gameplay which is slightly different from the video it used for training?

No, it implements a 3D FPS that's interactive, and renders each frame based on your input and a lot of memorized gameplay.

But is it playing the actual game or just making a interactive video of it?

Re: Diffusion models are real-time game engines

#260

Post by Workaccount2 »

wavemode wrote: post #190
vessenes wrote: post #2

So, this is surprising. Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected, which would be roughly ‘none’. Google here uses SD 1.4, as the core of the diffusion model, which is a nice reminder that open models are useful to even giant cloud monopolies. The two main things of note I took away from the summary were: 1) they got infinite training data using agents playing doom…

> Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected To temper this a bit, you may want to pay close attention to the demo videos. The player rarely backtracks, and for good reason - the few times the character does turn around and look back at something a second time, it has changed significantly (the most noticeable I think is the room with the grey wall and triangle sign…

I don't see this as something that would be hard to overcome. Sora for instance has already shown the ability for a diffusion model to maintain object permanence. Flux recently too has shown the ability to render the same person in many different poses or images.
Post reply on HN