Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Diffusion models are real-time game engines

gamengen.github.io

Post reply on HN
Print view View original

101–110 of 430 posts

Re: Diffusion models are real-time game engines

#101

Post by InDubioProRubio »

slashdave wrote: post #40
wkcheng wrote: post #9

It's insane that that this works, and that it works fast enough to render at 20 fps. It seems like they almost made a cross between a diffusion model and an RNN, since they had to encode the previous frames and actions and feed it into the model at each step. Abstractly, it's like the model is dreaming of a game that it played a lot of, and real time inputs just change the state of the dream. It makes me wonder if hu…

Image is 2D. Video is 3D. The mathematical extension is obvious. In this case, low resolution 2D (pixels), and the third dimension is just frame rate (discrete steps). So rather simple.

Video is also higher resolution, as the pixels flip for the high resolution world by moving through it. Swivelling your head without glasses, even the blurry world contains more information in the curve of pixelchange.

Re: Diffusion models are real-time game engines

#102

Post by agys »

qnleigh wrote: post #100

Could a similar scheme be used to drastically improve the visual quality of a video game? You would train the model on gameplay rendered at low and high quality (say with and without ray tracing, and with low and high density meshing), and try to get it to convert a quick render into something photorealistic on the fly. When things like DALL-E first came out, I was expecting something like the above to make it into m…

Isn't that what Nvidia’s Ray Reconstruction and DLSS (frame generation and upscaler) are doing, more or less?

Re: Diffusion models are real-time game engines

#103

Post by WithinReason »

Vecr wrote: post #34
zzanz wrote: post #3

The quest to run doom on everything continues. Technically speaking, isn't this the greatest possible anti-Doom, the Doom with the highest possible hardware requirement? I just find it funny that on a linear scale of hardware specification, Doom now finds itself on both ends.

It's the No-Doom.

Undoom?

Re: Diffusion models are real-time game engines

#104

Post by zbendefy »

nolist_policy wrote: post #95

Makes me wonder... If you stand still in front of a door so all past observations only contain that door, will the model teleport you to another level when opening the door?

I think some state is also being given (or if its not, it could be given) to the network, like 3d world position/orientation of the player, that could help the neural network anchor the player in the world.

Re: Diffusion models are real-time game engines

#106

Post by lIl-IIIl »

How does it know how many times it needs to shoot the zombie before it dies?

Most enemies have enough hit points to survive the first shot. If the model is only trained on the previous frame, it doesn't know how many times the enemy was already shot at.

From the video it seems like it is probability based - they may die right away or it might take way longer than it should.

I love how the player's health goes down when he stands in the radioactive green water.

In Doom the enemies fight with each other if they accidentally incur "friendly fire". It would be interesting to see it play out in this version.

Re: Diffusion models are real-time game engines

#107

Post by golol »

What I understand is the folloeing: If this works so well, why didn't we have good video generation much earlier? After diffusion models were seen to work the most obvious thing to do was to generate the next frame based on previous framrs but... it took 1-2 years for good video models to appear. For example compare Sora generating minecraft video versus this method generating minecraft video. Say in both cases the player is standing on a meadow with fee inputs and watching some pigs. In the Sora video you'd expect the typical glitched to appear, like erratic, sliding movement, overlapping legs, multiplication of pigs etc. Would these glitches not appear in the GameNGen video? Why?

Re: Diffusion models are real-time game engines

#108

Post by golol »

lIl-IIIl wrote: post #106

How does it know how many times it needs to shoot the zombie before it dies? Most enemies have enough hit points to survive the first shot. If the model is only trained on the previous frame, it doesn't know how many times the enemy was already shot at. From the video it seems like it is probability based - they may die right away or it might take way longer than it should. I love how the player's health goes down wh…

It gets a number of previous frames as input I think.

Re: Diffusion models are real-time game engines

#109

Post by Closi »

golol wrote: post #107

What I understand is the folloeing: If this works so well, why didn't we have good video generation much earlier? After diffusion models were seen to work the most obvious thing to do was to generate the next frame based on previous framrs but... it took 1-2 years for good video models to appear. For example compare Sora generating minecraft video versus this method generating minecraft video. Say in both cases the p…

Because video is much more difficult than images (it's lots of images that have to be consistent across time, with motion following laws of physics etc), and this is much more limited in terms of scope than pure arbitrary video generation.

Re: Diffusion models are real-time game engines

#110

Post by pantalaimon »

golol wrote: post #107

What I understand is the folloeing: If this works so well, why didn't we have good video generation much earlier? After diffusion models were seen to work the most obvious thing to do was to generate the next frame based on previous framrs but... it took 1-2 years for good video models to appear. For example compare Sora generating minecraft video versus this method generating minecraft video. Say in both cases the p…

I would have thought it is much easier to generate huge amounts of game footage for training, but as I understand this is not what was done here.
Post reply on HN