Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Diffusion models are real-time game engines

gamengen.github.io

Post reply on HN
Print view View original

141–150 of 430 posts

Re: Diffusion models are real-time game engines

#141

Post by meheleventyone »

lIl-IIIl wrote: post #106

How does it know how many times it needs to shoot the zombie before it dies? Most enemies have enough hit points to survive the first shot. If the model is only trained on the previous frame, it doesn't know how many times the enemy was already shot at. From the video it seems like it is probability based - they may die right away or it might take way longer than it should. I love how the player's health goes down wh…

> I love how the player's health goes down when he stands in the radioactive green water.

This is one of the bits that was weird to me, it doesn't work correctly. In the real game you take damage at a consistent rate, in the video the player doesn't and whether the player takes damage or not seems highly dependent on some factor that isn't whether or not the player is in the radioactive slime. My thought is that its learnt something else that correlates poorly.

Re: Diffusion models are real-time game engines

#142

Post by tobr »

snickmy wrote: post #129
godelski wrote: post #125

Doom system requirements: - 4 MB RAM - 12 MB disk space Stable diffusion v1 > 860M UNet and CLIP ViT-L/14 (540M) Checkpoint size: 4.27 Gb 7.7 GB (full EMA) Running on a TPU-v5e Peak compute per chip (bf16) 197 TFLOPs Peak compute per chip (Int8) 393 TFLOPs HBM2 capacity and bandwidth 16 GB, 819 GBps Interchip Interconnect BW 1600 Gbps This is quite impressive, especially considering the speed. But there's still a ton…

Those are valid points, but irrelevant for the context of this research. Yes, the computational cost is ridicolous compared to the original game, and yes, it lacks basic things like pre-computing, storing, etc. That said, you could assume that all that can be either done at the margin of this discovery OR over time will naturally improve OR will become less important as a blocker. The fact that you can model a sequen…

I suppose it also doesn't really matter what kinds of resources the game originally requires. The diffusion model isn't going to require twice as much memory just because the game does. Presumably you wouldn't even necessarily need to be able to render the original game in real time - I would imagine the basic technique would work even if you used a state of the Hollywood-quality offline renderer to render each input frame, and that the performance of the diffusion model would be similar?

Re: Diffusion models are real-time game engines

#143

Post by WithinReason »

nine_k wrote: post #134
vessenes wrote: post #2

So, this is surprising. Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected, which would be roughly ‘none’. Google here uses SD 1.4, as the core of the diffusion model, which is a nice reminder that open models are useful to even giant cloud monopolies. The two main things of note I took away from the summary were: 1) they got infinite training data using agents playing doom…

But it's not a game. It's a memory of a game video, predicting the next frame based on the few previous frames, like "I can imagine what happened next". I would call it the world's least efficient video compression. What I would like to see is the actual predictive strength, aka imagination, which I did not notice mentioned in the abstract. The model is trained on a set of classic maps. What would it do, given a few…

If it's trained on absolute player coordinates then it would likely just morph into the known map at those coordinates.

Re: Diffusion models are real-time game engines

#144

Post by Kiro »

richard___ wrote: post #71

Uhhh… demos would be more convincing with enemies and decreasing health

I see enemies and decreasing health on hit. But even if it lacked those, it seems like a pretty irrelevant nitpick that is completely underplaying what we're seeing here. The fact that this is even possible at all feels like science fiction.

Re: Diffusion models are real-time game engines

#145

Post by akie »

injidup wrote: post #68
kragen wrote: post #10

Earlier quoted context omitted.

here are some ideas: - you could build a non-real-time version of the game engine and use the neural net as a real-time approximation - you could edit videos shot in real life to have huds or whatever and train the neural net to simulate reality rather than doom. (this paper used 900 million frames which i think is about a year of video if it's 30fps, but maybe algorithmic improvements can cut the training requiremen…

Why games? I will train it on 1 years worth of me attending Microsoft teams meetings. Then I will go surfing.

Ready to pay for this

Re: Diffusion models are real-time game engines

#146

Post by nine_k »

WithinReason wrote: post #143
nine_k wrote: post #134

Earlier quoted context omitted.

But it's not a game. It's a memory of a game video, predicting the next frame based on the few previous frames, like "I can imagine what happened next". I would call it the world's least efficient video compression. What I would like to see is the actual predictive strength, aka imagination, which I did not notice mentioned in the abstract. The model is trained on a set of classic maps. What would it do, given a few…

If it's trained on absolute player coordinates then it would likely just morph into the known map at those coordinates.

But it's trained on the actual screen pixel data, AFAICT. It's literally a visual imagination model, not gameplay / geometry imagination model. They had to make special provisions to the pixel data on the HUD which by its nature different than the pictures of a 3D world.

Re: Diffusion models are real-time game engines

#147

Post by golol »

Closi wrote: post #109
golol wrote: post #107

What I understand is the folloeing: If this works so well, why didn't we have good video generation much earlier? After diffusion models were seen to work the most obvious thing to do was to generate the next frame based on previous framrs but... it took 1-2 years for good video models to appear. For example compare Sora generating minecraft video versus this method generating minecraft video. Say in both cases the p…

Because video is much more difficult than images (it's lots of images that have to be consistent across time, with motion following laws of physics etc), and this is much more limited in terms of scope than pure arbitrary video generation.

This misses the point, I'm comparing two methods of generating minecraft videos.

Re: Diffusion models are real-time game engines

#150

Post by birracerveza »

ynniv wrote: post #24
fngjdflmdflg wrote: post #8

Earlier quoted context omitted.

>Technically speaking, isn't this the greatest possible anti-Doom When I read this part I thought you were going to say because you're technically not running Doom at all. That is, instead of running Doom without Doom's original hardware/software environment (by porting it), you're running Doom without Doom itself.

It's dreaming Doom.

We made machines dream of Doom. Insane.
Post reply on HN