Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Diffusion models are real-time game engines

gamengen.github.io

Post reply on HN
Print view View original

91–100 of 430 posts

Re: Diffusion models are real-time game engines

#91

Post by quickestpoint »

lokimedes wrote: post #33
wkcheng wrote: post #9

It's insane that that this works, and that it works fast enough to render at 20 fps. It seems like they almost made a cross between a diffusion model and an RNN, since they had to encode the previous frames and actions and feed it into the model at each step. Abstractly, it's like the model is dreaming of a game that it played a lot of, and real time inputs just change the state of the dream. It makes me wonder if hu…

It makes good sense for humans to have this ability. If we flip the argument, and see the next frame as a hypothesis for what is expected as the outcome of the current frame, then comparing this "hypothesis" with what is sensed makes it easier to process the differences, rather than the totality of the sensory input. As Richard Dawkins recently put it in a podcast[1], our genes are great prediction machines, as their…

As Richard Dawkins theorized, would be more accurate and less LLM like :)

Re: Diffusion models are real-time game engines

#92

Post by troupo »

Key: "predicts next frame, recreates classic Doom". A game that was analyzed and documented to death. And the training included uncountable runs of Doom.

A game engine lets you create a new game, not predict the next frame of an existing and copiously documented one.

This is not a game engine.

Creating a new good game? Good luck with that.

Re: Diffusion models are real-time game engines

#93

Post by Jensson »

darrinm wrote: post #54
furyofantares wrote: post #49

Earlier quoted context omitted.

Short clips are the only way a human will make any errors determining which is which.

More relevant is if by _playing_ it they couldn’t tell which is which.

They obviously can within seconds, so it wouldn't be a result. Being able to generate gameplay that looks right even if it doesn't play right is one step.

Re: Diffusion models are real-time game engines

#94

Post by [deleted] »

toppy wrote: post #80
helloplanets wrote: post #69

So, any given sequence of inputs is rebuilt into a corresponding image, twenty times per second. I wonder how separate the game logic and the generated graphics are in the fully trained model. Given a sufficient enough separation between these two, couldn't you basically boil the game/input logic down to an abstract game template? Meaning, you could just output a hash that corresponds to a specific combination of inp…

I think you've just encoded the title of the paper

[deleted]

Re: Diffusion models are real-time game engines

#96

Post by SomewhatLikely »

refibrillator wrote: post #12

There is no text conditioning provided to the SD model because they removed it, but one can imagine a near future where text prompts are enough to create a fun new game! Yes they had to use RL to learn what DOOM looks like and how it works, but this doesn’t necessarily pose a chicken vs egg problem. In the same way that LLMs can write a novel story, despite only being trained on existing text. IMO one of the biggest…

Video games are gonna be wild in the near future. You could have one person talking to a model producing something that's on par with a AAA title from today. Imagine the 2d sidescroller boom on Steam but with immersive photorealistic 3d games with hyper-realistic physics (water flow, fire that spreads, tornados) and full deformability and buildability because the model is pretrained with real world videos. Your game is just a "style" that tweaks some priors on look, settings, and story.

Re: Diffusion models are real-time game engines

#97

Post by lukol »

I believe future game engines will be state machines with deterministic algorithms that can be reproduced at any time. However, rendering said state into visual / auditory / etc. experiences will be taken over by AI models.

This will also allow players to easily customize what they experience without changing the core game loop.

Re: Diffusion models are real-time game engines

#98

Post by jamilton »

I wonder if the MineRL (https://www.ijcai.org/proceedings/2019/0339.pdf and minerl.io) dataset would be sufficient to reproduce this work with Minecraft.

Any other similar existing datasets?

A really goofy way I can think of to get a bunch of data would be to get videos from youtube and try to detect keyboard sounds to determine what keys they're pressing.

Re: Diffusion models are real-time game engines

#99

Post by mistercheph »

energy123 wrote: post #35
mlsu wrote: post #31

Earlier quoted context omitted.

With enough computation, your neural net weights would converge to some very compressed latent representation of the source code of DOOM. Maybe smaller even than the source code itself? Someone in the field could probably correct me on that. At which point, you effectively would be interpolating in latent space through the source code to actually "render" the game. You'd have an entire latent space computer, with an…

The source code lacks information required to render the game. Textures for example.

That’s just an artifact of the language we use to describe an implementation detail, in the sense GP means it, the data payload bits are not essentially distinct from the executable instruction bits

Re: Diffusion models are real-time game engines

#100

Post by qnleigh »

Could a similar scheme be used to drastically improve the visual quality of a video game? You would train the model on gameplay rendered at low and high quality (say with and without ray tracing, and with low and high density meshing), and try to get it to convert a quick render into something photorealistic on the fly.

When things like DALL-E first came out, I was expecting something like the above to make it into mainstream games within a few years. But that was either too optimistic or I'm not up to speed on this sort of thing.

Post reply on HN