Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Diffusion models are real-time game engines

gamengen.github.io

Post reply on HN
Print view View original

211–220 of 430 posts

Re: Diffusion models are real-time game engines

#211

Post by [deleted] »

mensetmanusman wrote: post #200
rvnx wrote: post #191

Earlier quoted context omitted.

The concept is that if you train a Diffusion model by feeding all the possible frames seen in the game. The training was over almost 1 billion frames, 20 days of full-time play-time, taking a screenshot of every single inch of the map. Now you show him N frames as input, and ask it "give me frame N+1", then it gives you the frame n. N+1 back based on how it was originally seen during training. But it is not frame N+1…

Research is the acquisition of knowledge that may or may not have practical applications. They succeeded in the research, gained knowledge, and might be able to do something awesome with it. It’s a success even if they don’t sell anything.

[deleted]

Re: Diffusion models are real-time game engines

#212

Post by codeflo »

wavemode wrote: post #190
vessenes wrote: post #2

So, this is surprising. Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected, which would be roughly ‘none’. Google here uses SD 1.4, as the core of the diffusion model, which is a nice reminder that open models are useful to even giant cloud monopolies. The two main things of note I took away from the summary were: 1) they got infinite training data using agents playing doom…

> Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected To temper this a bit, you may want to pay close attention to the demo videos. The player rarely backtracks, and for good reason - the few times the character does turn around and look back at something a second time, it has changed significantly (the most noticeable I think is the room with the grey wall and triangle sign…

Even purely going forward, specks on wall textures morph into opponents and so on. All the diffusion-generated videos I’ve seen so far have this kind of unsettling feature.

Re: Diffusion models are real-time game engines

#213

Post by debo_ »

mensetmanusman wrote: post #199
wavemode wrote: post #190

Earlier quoted context omitted.

> Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected To temper this a bit, you may want to pay close attention to the demo videos. The player rarely backtracks, and for good reason - the few times the character does turn around and look back at something a second time, it has changed significantly (the most noticeable I think is the room with the grey wall and triangle sign…

That is kind of cool though, I would play like being lost in a dream. If on the backend you could record the level layouts in memory you could have exploration teams that try to find new areas to explore.

It would be cool for dream sequences in games to feel more like dreams. This is probably an expensive way to do it, but it would be neat!

Re: Diffusion models are real-time game engines

#214

Post by Sharlin »

hypertele-Xii wrote: post #198
SeanAnderson wrote: post #64

Earlier quoted context omitted.

I think you're mistaken. The abstract says it's interactive, "We present GameNGen, the first game engine powered entirely by a neural model that enables real-time interaction" Further - "a diffusion model is trained to produce the next frame, conditioned on the sequence of past frames and actions." specifically "and actions" User input is being fed into this system and subsequent frames take that into account. The us…

Then why do monsters become blurry smudgy messes when shot? That looks like a video compression artifact of a neural network attempting to replicate low-structure image (source material contains guts exploding, very un-structured visual).

Uh, maybe because monster death animations make up a small part of the training material (ie. gameplay) so the model has not learned to reproduce them very well?

There cannot be "video compression artifacts" because it hasn’t even seen any compressed video during training, as far as I can see.

Seriously, how is this even a discussion? The article is clear that the novel thing is that this is real-time frame generation conditioned on the previous frame(s) AND player actions. Just generating video would be nothing new.

Re: Diffusion models are real-time game engines

#216

Post by Sharlin »

nine_k wrote: post #134
vessenes wrote: post #2

So, this is surprising. Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected, which would be roughly ‘none’. Google here uses SD 1.4, as the core of the diffusion model, which is a nice reminder that open models are useful to even giant cloud monopolies. The two main things of note I took away from the summary were: 1) they got infinite training data using agents playing doom…

But it's not a game. It's a memory of a game video, predicting the next frame based on the few previous frames, like "I can imagine what happened next". I would call it the world's least efficient video compression. What I would like to see is the actual predictive strength, aka imagination, which I did not notice mentioned in the abstract. The model is trained on a set of classic maps. What would it do, given a few…

No, it’s predicting the next frame conditioned on past frames AND player actions! This is clear from the article. Mere video generation would be nothing new.

Re: Diffusion models are real-time game engines

#217

Post by whiteboardr »

wavemode wrote: post #190
vessenes wrote: post #2

So, this is surprising. Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected, which would be roughly ‘none’. Google here uses SD 1.4, as the core of the diffusion model, which is a nice reminder that open models are useful to even giant cloud monopolies. The two main things of note I took away from the summary were: 1) they got infinite training data using agents playing doom…

> Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected To temper this a bit, you may want to pay close attention to the demo videos. The player rarely backtracks, and for good reason - the few times the character does turn around and look back at something a second time, it has changed significantly (the most noticeable I think is the room with the grey wall and triangle sign…

But does it need to be frame-based?

What if you combine this with an engine in parallel that provides all geometry including characters and objects with their respective behavior, recording changes made through interactions the other model generates, talking back to it?

A dialogue between two parties with different functionality so to speak.

(Non technical person here - just fantasizing)

Re: Diffusion models are real-time game engines

#218

Post by OskarS »

rvnx wrote: post #208
jetrink wrote: post #197

Earlier quoted context omitted.

I don't think you've understood the project completely. The model accepts player input, so frame 601 could be quite different if the player decided to turn left rather than right, or chose that moment to fire at an exploding barrel.

1 billion frames in memory... With such dataset, you have seen practically all realistic possibilities in the short-term. If it would be able to invent action and maps and let the user play "infinite doom", then it would be very different (and impressive!).

> 1 billion frames in memory... With such dataset, you have seen practically all realistic possibilities in the short-term.

I mean... no? Not even close? Multiply the number of game states with the number of inputs at any given frame gives you a number vastly bigger than 1 billion, not even comparable. Even with 20 days of play time to train no, it's entirely likely that at no point did someone stop at a certain location and look to the left from that angle. They might have done from similar angles, but the model then has to reconstruct some sense of the geometry of the level to synthesize the frame. They might also not have arrived there from the same direction, which again the model needs some smarts to understand.

I get your point, it's very overtrained on these particular levels of Doom, which means you might as well just play Doom. But this is not a hash table lookup we're talking about, it's pretty impressive work.

Re: Diffusion models are real-time game engines

#219

Post by mind-blight »

quickestpoint wrote: post #85
stevenhuang wrote: post #67

Earlier quoted context omitted.

> It makes me wonder if humans are just next moment prediction machines, with just a little bit more memory built in. Yup, see https://en.wikipedia.org/wiki/Predictive_coding

Umm, that’s a theory.

So are gravity and friction. I don't know how well tested or accepted it is, but being just a theory doesn't tell you much about how true it is without more info

Re: Diffusion models are real-time game engines

#220

Post by GaggiX »

gwbas1c wrote: post #215

Am I the only one who thinks this is faked? It's not that hard to fake something like this: Just make a video of DOSBox with DOOM running inside of it, and then compress it with settings that will result in compression artifacts.

>Am I the only one who thinks this is faked?

Yes.

Post reply on HN