Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Diffusion models are real-time game engines

gamengen.github.io

Post reply on HN
Print view View original

381–390 of 430 posts

Re: Diffusion models are real-time game engines

#381

Post by slashdave »

InDubioProRubio wrote: post #101
slashdave wrote: post #40

Earlier quoted context omitted.

Image is 2D. Video is 3D. The mathematical extension is obvious. In this case, low resolution 2D (pixels), and the third dimension is just frame rate (discrete steps). So rather simple.

Video is also higher resolution, as the pixels flip for the high resolution world by moving through it. Swivelling your head without glasses, even the blurry world contains more information in the curve of pixelchange.

Correct, for the sprites. However, the walls in Doom are texture mapped, and so have the same issue as videos. Interesting, though, because I assume the antialiasing is something approximate, given the extreme demands on CPUs of the era.

Re: Diffusion models are real-time game engines

#382

Post by Workaccount2 »

Jensson wrote: post #324
Workaccount2 wrote: post #295

Earlier quoted context omitted.

I don't know of an example (not to say it doesn't exist) but the problem is fundamentally the same as things moving out of sight/out of frame and coming back again.

> the problem is fundamentally the same as things moving out of sight/out of frame and coming back again Maybe it is, but doing that with the entire scene instead of just a small part of it makes the problem massively harder, as the model needs to grow exponentially to remember more things. It isn't something that we will manage anytime soon, maybe 10-20 years with current architecture and same compute progress. Then…

10 to 20 years sounds wildly pessimistic

In this sora video the dragon covers half the scene, and its basically identical when it is revealed again ~5 seconds later, or about 150 frames later. The is lots of evidence (and some studies) that these models are in fact building internal world models.

https://www.youtube.com/watch?v=LXJ-yLiktDU

Buckle in, the train is moving way faster. I don't think there would be much surprise if this is solved in the next few generations of video generators. The first generation is already doing very well.

Re: Diffusion models are real-time game engines

#383

Post by bangaladore »

wkcheng wrote: post #9

It's insane that that this works, and that it works fast enough to render at 20 fps. It seems like they almost made a cross between a diffusion model and an RNN, since they had to encode the previous frames and actions and feed it into the model at each step. Abstractly, it's like the model is dreaming of a game that it played a lot of, and real time inputs just change the state of the dream. It makes me wonder if hu…

> It's insane that that this works, and that it works fast enough to render at 20 fps.

It is running on an entire v5 TPU (https://cloud.google.com/blog/products/ai-machine-learning/i...)

It's unclear how that compares to a high-end consumer GPU like a 3090, but they seem to have similar INT8 TFLOPS. The TPU has less memory (16 vs. 24), and I'm unsure of the other specs.

Something doesn't add up, in my opinion, though. SD usually takes (at minimum) seconds to produce a high-quality result on a 3090, so I can't comprehend how they are like 2 orders of magnitudes faster—indicating that the TPU vastly outperforms a GPU for this task. They seem to be producing low-res (320x240) images, but it still seems too fast.

Re: Diffusion models are real-time game engines

#384

Post by Jensson »

Workaccount2 wrote: post #382
Jensson wrote: post #324

Earlier quoted context omitted.

> the problem is fundamentally the same as things moving out of sight/out of frame and coming back again Maybe it is, but doing that with the entire scene instead of just a small part of it makes the problem massively harder, as the model needs to grow exponentially to remember more things. It isn't something that we will manage anytime soon, maybe 10-20 years with current architecture and same compute progress. Then…

10 to 20 years sounds wildly pessimistic In this sora video the dragon covers half the scene, and its basically identical when it is revealed again ~5 seconds later, or about 150 frames later. The is lots of evidence (and some studies) that these models are in fact building internal world models. https://www.youtube.com/watch?v=LXJ-yLiktDU Buckle in, the train is moving way faster. I don't think there would be much s…

Did you watch the video, it is completely different after the dragon goes past? Its still a flag there, but everything else changed. Even the stores in the background changed, the mass of people is completely different with no hint of anyone moving there etc.

You always get this from AI enthusiast, they come and post "proof" that disproves their own point.

Re: Diffusion models are real-time game engines

#385

Post by crooked-v »

lawlessone wrote: post #321
ValentinA23 wrote: post #285

Earlier quoted context omitted.

What you're pointing at mirrors the same kind of limitation in using LLMs for role-play/interactive fictions.

Maybe a hybrid approach would work. Certain things like inventory being stored as variables, lists etc. Wouldn't be as pure though.

Give it state by having a rendered-but-offscreen pixel area that's fed back in as byte data for the next frame.

Re: Diffusion models are real-time game engines

#386

Post by HappMacDonald »

whiteboardr wrote: post #267
beepbooptheory wrote: post #245

Earlier quoted context omitted.

What would the model provide if not what we see on the screen?

The environment and everything in it. “Everything” would mean all objects and the elements they’re made of, their rules on how they interact and decay. A modularized ecosystem i guess, comprised of “sub-systems” of sorts. The other model, that provides all interaction (cause for effect) could either be run artificially or be used interactively by a human - opening up the possibility for being a tree : ) This all woul…

So you're basically just talking about upgrading "enemy AI" to a more complex form of AI :)

Re: Diffusion models are real-time game engines

#387

Post by HappMacDonald »

Jensson wrote: post #384
Workaccount2 wrote: post #382

Earlier quoted context omitted.

10 to 20 years sounds wildly pessimistic In this sora video the dragon covers half the scene, and its basically identical when it is revealed again ~5 seconds later, or about 150 frames later. The is lots of evidence (and some studies) that these models are in fact building internal world models. https://www.youtube.com/watch?v=LXJ-yLiktDU Buckle in, the train is moving way faster. I don't think there would be much s…

Did you watch the video, it is completely different after the dragon goes past? Its still a flag there, but everything else changed. Even the stores in the background changed, the mass of people is completely different with no hint of anyone moving there etc. You always get this from AI enthusiast, they come and post "proof" that disproves their own point.

I'm not GP, but running over that video I'm actually having a hard time finding any detail present before the dragon obscures them not either exit frame right when the camera pans left slightly near the end or not re-appear with reasonably crisp detail after the dragon gets out of the way.

Most of the mob of people are indistinct, but there is a woman in a lime green coat who is visible, and then obstructed by the dragon twice (beard and ribbon) and reappears fine. Unfortunately when dragon fully moves past she has been lost to frame right.

There is another person in black holding a red satchel which is visible both before and after the dragon has passed.

Nothing about the storefronts appear to change. The complex sign full of Chinese text (which might be gibberish text: it's highly stylized and I don't know Chinese) appears to survive the dragon passing without even any changes to the individual ideograms.

There is also a red box shaped like a Chinese paper lantern with a single gold ideogram on it at the store entrance which spends most of the video obscured by the dragon and is still in the same location after it passes (though video artifacting makes it more challenging to verify that that ideogram is unchanged it certainly does not appear substantially different)

What detail are you seeing that is different before and after the obstruction?

Re: Diffusion models are real-time game engines

#388

Post by dvngnt_ »

dTal wrote: post #165
godelski wrote: post #125

Doom system requirements: - 4 MB RAM - 12 MB disk space Stable diffusion v1 > 860M UNet and CLIP ViT-L/14 (540M) Checkpoint size: 4.27 Gb 7.7 GB (full EMA) Running on a TPU-v5e Peak compute per chip (bf16) 197 TFLOPs Peak compute per chip (Int8) 393 TFLOPs HBM2 capacity and bandwidth 16 GB, 819 GBps Interchip Interconnect BW 1600 Gbps This is quite impressive, especially considering the speed. But there's still a ton…

>What's also interesting about this work is it's basically saying you can rip a game if you're willing to "play" (automate) it enough times and spend a lot more on storage and compute That's the least of it. It means you can generate a game from real footage. Want a perfect flight sim? Put a GoPro in the cockpit of every airliner for a year.

wouldnt make more sense to train using microsoft flight simulator the same way they did DOOM, but im not sure what the point is if the game already exists

Re: Diffusion models are real-time game engines

#389

Post by elwell »

ynniv wrote: post #24
fngjdflmdflg wrote: post #8

Earlier quoted context omitted.

>Technically speaking, isn't this the greatest possible anti-Doom When I read this part I thought you were going to say because you're technically not running Doom at all. That is, instead of running Doom without Doom's original hardware/software environment (by porting it), you're running Doom without Doom itself.

It's dreaming Doom.

Droom

Re: Diffusion models are real-time game engines

#390

Post by foota »

Chance-Device wrote: post #370
foota wrote: post #354

Earlier quoted context omitted.

I wonder if they could somehow feed in a trained Gaussian splats model to this to get better images? Since the splats are specifically designed for rendering it seems like it would be an efficient way for the image model to learn the geometry without having to encode it on the image model itself.

I’m not sure how that would help vs just training the model with the conditionings described in the paper. I’m not very familiar with Gaussian splats models, but aren’t they just a way of constructing images using multiple superimposed parameterized Gaussian distributions, sort of like the Fourier series does with waveforms using sine and cosine waves? I’m not seeing how that would apply here but I’d be interested in…

I'm not certain where it would fit in, but my thinking is this.

There's been a bunch of work on making splats efficient and good at representing geometry. Reading more, perhaps NERFs would be a better fit, since they're an actual neutral network.

My thinking is that if you trained a NERF ahead of time to represent the geometry and layout of the levels, and plug that in to the diffusion model (as a part of computing the latents, and then also on the other side so it can be used to improve the rendering) then the diffusion model could focus on learning how actions manipulate the world without having to learn the geometry representation.

Post reply on HN