Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Diffusion models are real-time game engines

gamengen.github.io

Post reply on HN
Print view View original

301–310 of 430 posts

Re: Diffusion models are real-time game engines

#301

Post by dartos »

wkcheng wrote: post #9

It's insane that that this works, and that it works fast enough to render at 20 fps. It seems like they almost made a cross between a diffusion model and an RNN, since they had to encode the previous frames and actions and feed it into the model at each step. Abstractly, it's like the model is dreaming of a game that it played a lot of, and real time inputs just change the state of the dream. It makes me wonder if hu…

> It makes me wonder if humans are just next moment prediction machines, with just a little bit more memory built in.

This, to me, seems extremely reductionist. Like you start with AI and work backwards until you frame all cognition as next something predictors.

It’s just the stochastic parrot argument again.

Re: Diffusion models are real-time game engines

#302

Post by [deleted] »

Workaccount2 wrote: post #292
aithrowaway1987 wrote: post #81

Earlier quoted context omitted.

Have you ever played a video game? This is unbelievably depressing. This is a future where games like Slay the Spire, with a unique art style and innovative gameplay simply are not being made. Not to mention this childish nonsense about "forget they're playing a game," as if every game needs to be lifelike VR and there's no room for stylization or imagination. I am worried for the future that people think they want t…

The problem is quite the opposite, that AI will be able to generate games so many game with so many play styles that it will totally dilute the value of all games. Compare it to music gen algo's that can now produce music that is 100% indiscernible from generic crappy music. Which is insane given that 5 years ago it could maybe create the sound of something that maybe someone would describe as "sort of guitar-like".…

[deleted]

Re: Diffusion models are real-time game engines

#303

Post by pradn »

vessenes wrote: post #2

So, this is surprising. Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected, which would be roughly ‘none’. Google here uses SD 1.4, as the core of the diffusion model, which is a nice reminder that open models are useful to even giant cloud monopolies. The two main things of note I took away from the summary were: 1) they got infinite training data using agents playing doom…

> Google here uses SD 1.4, as the core of the diffusion model, which is a nice reminder that open models are useful to even giant cloud monopolies.

A mistake people make all the time is that massive companies will put all their resources toward every project. This paper was written by four co-authors. They probably got a good amount of resources, but they still had to share in the pool allocated to their research department.

Even Google only has one Gemini (in a few versions).

Re: Diffusion models are real-time game engines

#304

Post by phh »

dTal wrote: post #165
godelski wrote: post #125

Doom system requirements: - 4 MB RAM - 12 MB disk space Stable diffusion v1 > 860M UNet and CLIP ViT-L/14 (540M) Checkpoint size: 4.27 Gb 7.7 GB (full EMA) Running on a TPU-v5e Peak compute per chip (bf16) 197 TFLOPs Peak compute per chip (Int8) 393 TFLOPs HBM2 capacity and bandwidth 16 GB, 819 GBps Interchip Interconnect BW 1600 Gbps This is quite impressive, especially considering the speed. But there's still a ton…

>What's also interesting about this work is it's basically saying you can rip a game if you're willing to "play" (automate) it enough times and spend a lot more on storage and compute That's the least of it. It means you can generate a game from real footage. Want a perfect flight sim? Put a GoPro in the cockpit of every airliner for a year.

> Want a perfect flight sim? Put a GoPro in the cockpit of every airliner for a year.

I guess that's the occasion to remind that ML is splendid at interpolating, but extrapolating, maybe don't keep your hopes too high.

Namely, to have a "perfect flight sim" using GoPros, you'll need to record hundreds of stalls and crashs.

Re: Diffusion models are real-time game engines

#305

Post by stnmtn »

darby_nine wrote: post #238
kqr wrote: post #232

Earlier quoted context omitted.

> even rules which are not visible on-screen. If a rule was changed but it's never visible on the screen, did it really change? > It simply generated frames and the appearance of play mechanics from a game it sampled (which humans created). Simply?! I understand it's mechanically trivial but the fact that it's compressed such a rich conditional distribution seems far from simple to me.

> Simply?! I understand it's mechanically trivial but the fact that it's compressed such a rich conditional distribution seems far from simple to me. It's much simpler than actually creating a game....

If someone told you 10 years ago that they were going to create something where you could play a whole new level of Doom, without them writing a single line of game logic/rendering code, would you say that that is simpler than creating a demo by writing the game themselves?

Re: Diffusion models are real-time game engines

#306

Post by GaggiX »

meheleventyone wrote: post #299
GaggiX wrote: post #271

Earlier quoted context omitted.

>I mean doing an actual analysis of how well the simulation matches the ground truth of the game. Do you mean the PSNR and LPIPS metrics used in paper?

No, I think I've been pretty clear that I'm interested in how mechanically sound the simulation is. Also those measures are over an even shorter duration so even less relevant to how coherent it is at real game scales.

How should this be concretely evaluated and measured? A vibe check?

Re: Diffusion models are real-time game engines

#307

Post by kridsdale1 »

TeMPOraL wrote: post #263
wavemode wrote: post #234

Earlier quoted context omitted.

Yes it definitely is very good for simulating gameplay footage, don't get me wrong. Its input for predicting the next frame is not just the previous frame, it has access to a whole sequence of prior frames. But to say the model is simulating actual gameplay (i.e. that a person could actually play Doom in this) is far fetched. It's definitely great that the model was able to remember that the gray wall was still there…

> it's untenable for actual gameplay that the wall completely changed location and orientation. It would in an SCP-themed game. Or dreamscape/Inception themed one. Hell, "you're trapped in Doom-like dreamscape, escape before you lose your mind" is a very interesting pitch for a game. Basically take this Doom thing and make walking though a specific, unique-looking doorway from the original game to be the victory cond…

Check out the actual modern DOOM WAD MyHouse which implements these ideas. It totally breaks our preconceptions of what the DOOM engine is capable of.

https://en.wikipedia.org/wiki/MyHouse.wad

Re: Diffusion models are real-time game engines

#308

Post by famouswaffles »

dabochen wrote: post #248

So there is no interactivity, but the generated content is not the exact view in the training data, is this the correct understanding? If so, is it more like imagination/hallucination rather than rendering?

It's conditioned on previous frames AND player actions so it's interactive.

Re: Diffusion models are real-time game engines

#309

Post by rrnechmech »

> To mitigate auto-regressive drift during inference, we corrupt context frames by adding Gaussian noise to encoded frames during training. This allows the network to correct information sampled in previous frames, and we found it to be critical for preserving visual stability over long time periods.

I get this (mostly). But would any kind soul care to elaborate on this? What is this "drift" they are trying to avoid and how does (AFAIU) adding noise help?

Re: Diffusion models are real-time game engines

#310

Post by slashdave »

SeanAnderson wrote: post #64
slashdave wrote: post #56

Earlier quoted context omitted.

It is just video. There are no external interactions. Heck, it is far simpler than video, because the point of view and frame is fixed.

I think you're mistaken. The abstract says it's interactive, "We present GameNGen, the first game engine powered entirely by a neural model that enables real-time interaction" Further - "a diffusion model is trained to produce the next frame, conditioned on the sequence of past frames and actions." specifically "and actions" User input is being fed into this system and subsequent frames take that into account. The us…

No, I am not. The interaction is part of the training, and is used during inference, but it is not including during the process of generation.
Post reply on HN