Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Diffusion models are real-time game engines

gamengen.github.io

Post reply on HN
Print view View original

81–90 of 430 posts

Re: Diffusion models are real-time game engines

#81

Post by aithrowaway1987 »

w_for_wumbo wrote: post #13
kragen wrote: post #10

Earlier quoted context omitted.

here are some ideas: - you could build a non-real-time version of the game engine and use the neural net as a real-time approximation - you could edit videos shot in real life to have huds or whatever and train the neural net to simulate reality rather than doom. (this paper used 900 million frames which i think is about a year of video if it's 30fps, but maybe algorithmic improvements can cut the training requiremen…

That feels like the endgame of video game generation. You select an art style, a video and the type of game you'd like to play. The game is then generated in real-time responding to each action with respect to the existing rule engine. I imagine a game like that could get so convincing in its details and immersiveness that one could forget they're playing a game.

Have you ever played a video game? This is unbelievably depressing. This is a future where games like Slay the Spire, with a unique art style and innovative gameplay simply are not being made.

Not to mention this childish nonsense about "forget they're playing a game," as if every game needs to be lifelike VR and there's no room for stylization or imagination. I am worried for the future that people think they want these things.

Re: Diffusion models are real-time game engines

#82

Post by nopakos »

SeanAnderson wrote: post #64
slashdave wrote: post #56

Earlier quoted context omitted.

It is just video. There are no external interactions. Heck, it is far simpler than video, because the point of view and frame is fixed.

I think you're mistaken. The abstract says it's interactive, "We present GameNGen, the first game engine powered entirely by a neural model that enables real-time interaction" Further - "a diffusion model is trained to produce the next frame, conditioned on the sequence of past frames and actions." specifically "and actions" User input is being fed into this system and subsequent frames take that into account. The us…

Maybe it's so advanced, it knows the players' next moves, so it is a video!

Re: Diffusion models are real-time game engines

#83

Post by Jensson »

mlsu wrote: post #31
refibrillator wrote: post #12

There is no text conditioning provided to the SD model because they removed it, but one can imagine a near future where text prompts are enough to create a fun new game! Yes they had to use RL to learn what DOOM looks like and how it works, but this doesn’t necessarily pose a chicken vs egg problem. In the same way that LLMs can write a novel story, despite only being trained on existing text. IMO one of the biggest…

With enough computation, your neural net weights would converge to some very compressed latent representation of the source code of DOOM. Maybe smaller even than the source code itself? Someone in the field could probably correct me on that. At which point, you effectively would be interpolating in latent space through the source code to actually "render" the game. You'd have an entire latent space computer, with an…

> With enough computation, your neural net weights would converge to some very compressed latent representation of the source code of DOOM. Maybe smaller even than the source code itself? Someone in the field could probably correct me on that.

Neural nets are not guaranteed to converge to anything even remotely optimal, so no that isn't how it works. Also even though neural nets can approximate any function they usually can't do it in a time or space efficient manner, resulting in much larger programs than the human written code.

Re: Diffusion models are real-time game engines

#84

Post by hoseja »

TeMPOraL wrote: post #76
energy123 wrote: post #35

Earlier quoted context omitted.

The source code lacks information required to render the game. Textures for example.

Obviously assets would get encoded too, in some form. Not necessarily corresponding to the original bitmaps, if the game does some consistent post-processing, the encoded thing would more likely be (equivalent to) the post-processed state.

Finally, the AI superoptimizing compiler.

Re: Diffusion models are real-time game engines

#85

Post by quickestpoint »

stevenhuang wrote: post #67
wkcheng wrote: post #9

It's insane that that this works, and that it works fast enough to render at 20 fps. It seems like they almost made a cross between a diffusion model and an RNN, since they had to encode the previous frames and actions and feed it into the model at each step. Abstractly, it's like the model is dreaming of a game that it played a lot of, and real time inputs just change the state of the dream. It makes me wonder if hu…

> It makes me wonder if humans are just next moment prediction machines, with just a little bit more memory built in. Yup, see https://en.wikipedia.org/wiki/Predictive_coding

Umm, that’s a theory.

Re: Diffusion models are real-time game engines

#86

Post by troupo »

refibrillator wrote: post #12

There is no text conditioning provided to the SD model because they removed it, but one can imagine a near future where text prompts are enough to create a fun new game! Yes they had to use RL to learn what DOOM looks like and how it works, but this doesn’t necessarily pose a chicken vs egg problem. In the same way that LLMs can write a novel story, despite only being trained on existing text. IMO one of the biggest…

> one can imagine a near future where text prompts are enough to create a fun new game

Sit down and write down a text prompt for a "fun new game". You can start with something relatively simple like a Mario-like platformer.

By page 300, when you're about halfway through describing what you mean, you might understand why this is wishful thinking

Re: Diffusion models are real-time game engines

#87

Post by ukuina »

sitkack wrote: post #55

What most programmers don't understand, that in the very near future, the entire application will be delivered by an AI model, no source, no text, just connect to the app over RDP. The whole app will be created by example, the app developer will train the app like a dog trainer trains a dog.

So... https://websim.ai except over pixels instead of in your browser?

Re: Diffusion models are real-time game engines

#88

Post by troupo »

w_for_wumbo wrote: post #13
kragen wrote: post #10

Earlier quoted context omitted.

here are some ideas: - you could build a non-real-time version of the game engine and use the neural net as a real-time approximation - you could edit videos shot in real life to have huds or whatever and train the neural net to simulate reality rather than doom. (this paper used 900 million frames which i think is about a year of video if it's 30fps, but maybe algorithmic improvements can cut the training requiremen…

That feels like the endgame of video game generation. You select an art style, a video and the type of game you'd like to play. The game is then generated in real-time responding to each action with respect to the existing rule engine. I imagine a game like that could get so convincing in its details and immersiveness that one could forget they're playing a game.

There are thousands of games that mimic each other, and only a handful of them are any good.

What makes you think a mechanical "predict next frame based on existing games" will be any good?

Re: Diffusion models are real-time game engines

#89

Post by Jensson »

arduinomancer wrote: post #42

How does the model “remember” the whole state of the world? Like if I kill an enemy in some room and walk all the way across the map and come back, would the body still be there?

It doesn't even remember the state of the game you look at. Doors spawning right in front of you, particle effects turning into enemies mid flight etc, so just regular gen AI issues.

Edit: Can see this in the first 10 seconds of the first video under "Full Gameplay Videos", stairs turning to corridor turning to closed door for no reason without looking away.

Re: Diffusion models are real-time game engines

#90

Post by cubefox »

mo_42 wrote: post #63

An implementation of the game engine in the model itself is theoretically the most accurate solution for predicting the next frame. I'm wondering when people will apply this to other areas like the real world. Would it learn the game engine of the universe (ie physics)?

A popular theory in neuroscience is that this is what the brain does:

https://slatestarcodex.com/2017/09/05/book-review-surfing-un...

It's called predictive coding. By trying to predict sensory stimuli, the brain creates a simplified model of the world, including common sense physics. Yann LeCun says that this is a major key to AGI. Another one is effective planning.

But while current predictive models (autoregressive LLMs) work well on text, they don't work well on video data, because of the large outcome space. In an LLM, text prediction boils down to a probability distribution over a few thousand possible next tokens, while there are several orders of magnitude more possible "next frames" in a video. Diffusion models work better on video data, but they are not inherently predictive like causal LLMs. Apparently this new Doom model made some progress on that front though.

Post reply on HN