Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Diffusion models are real-time game engines

gamengen.github.io

Post reply on HN
Print view View original

271–280 of 430 posts

Re: Diffusion models are real-time game engines

#271

Post by GaggiX »

meheleventyone wrote: post #250
GaggiX wrote: post #189

Earlier quoted context omitted.

>rather than sitting comparable situations next to one another in the game and simulation then analyzing them. That's literally how the human rating was setup if you read the paper.

I think you misunderstand me. I don't mean a snap evaluation and deciding between two very-short competing videos which is what the participants were doing. I mean doing an actual analysis of how well the simulation matches the ground truth of the game. What I'd posit is that it's not actually a very good replication of the game but very good a replicating short clips that almost look like the game and the short time…

>I mean doing an actual analysis of how well the simulation matches the ground truth of the game.

Do you mean the PSNR and LPIPS metrics used in paper?

Re: Diffusion models are real-time game engines

#272

Post by lawlessone »

dewarrn1 wrote: post #221
wavemode wrote: post #190

Earlier quoted context omitted.

> Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected To temper this a bit, you may want to pay close attention to the demo videos. The player rarely backtracks, and for good reason - the few times the character does turn around and look back at something a second time, it has changed significantly (the most noticeable I think is the room with the grey wall and triangle sign…

Great observation. And not entirely unlike normal human visual perception which is notoriously vulnerable to missing highly salient information; I'm reminded of the "gorillas in our midst" work by Dan Simons and Christopher Chabris [0]. [0]: https://en.wikipedia.org/wiki/Inattentional_blindness#Invisi...

I reminds me of dreaming. When you do something and turn back to check it has turned into something completely different.

edit: someone should train it on MyHouse.wad

Re: Diffusion models are real-time game engines

#273

Post by TeMPOraL »

nine_k wrote: post #134
vessenes wrote: post #2

So, this is surprising. Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected, which would be roughly ‘none’. Google here uses SD 1.4, as the core of the diffusion model, which is a nice reminder that open models are useful to even giant cloud monopolies. The two main things of note I took away from the summary were: 1) they got infinite training data using agents playing doom…

But it's not a game. It's a memory of a game video, predicting the next frame based on the few previous frames, like "I can imagine what happened next". I would call it the world's least efficient video compression. What I would like to see is the actual predictive strength, aka imagination, which I did not notice mentioned in the abstract. The model is trained on a set of classic maps. What would it do, given a few…

It's a memory of a video looped to controls, so frame 1 is "I wonder how would it look if the player pressed D instead of W", then the frame 2 is based on frame 1, etc. and couple frames in, it's already not remembering, but imagining the gameplay on the fly. It's not prerecorded, it responds to inputs during generation. That's what makes it a game engine.

Re: Diffusion models are real-time game engines

#274

Post by Workaccount2 »

sharpshadow wrote: post #259
TeMPOraL wrote: post #247

Earlier quoted context omitted.

No, it implements a 3D FPS that's interactive, and renders each frame based on your input and a lot of memorized gameplay.

But is it playing the actual game or just making a interactive video of it?

What is the difference?

Re: Diffusion models are real-time game engines

#275

Post by drowsspa »

lokimedes wrote: post #171
dbspin wrote: post #127

Earlier quoted context omitted.

Worth noting that aphantasia doesn't necessarily extend to dreams. Anecdotally - I have pretty severe aphantasia (I can conjure milisecond glimpses of barely tangible imagery that I can't quite perceive before it's gone - but only since learning that visualisation wasn't a linguistic metaphor). I can't really simulate object rotation. I can't really 'picture' how things will look before they're drawn / built etc. How…

I fabulate about this in another comment below: > Many people with aphantasia reports being able to visualize in their dreams, meaning that they don't lack the ability to generate visuals. So it may be that the [aphantasia] brain has an affinity to rely on the abstract representation when "thinking", while dreaming still uses the "stable diffusion mode". (I obviously don't know what I'm talking about, just a fellow a…

Yeah. In my head it's like I'm manipulating SVG paths instead of raw pixels

Re: Diffusion models are real-time game engines

#276

Post by wrsh07 »

mensetmanusman wrote: post #209
wkcheng wrote: post #9

It's insane that that this works, and that it works fast enough to render at 20 fps. It seems like they almost made a cross between a diffusion model and an RNN, since they had to encode the previous frames and actions and feed it into the model at each step. Abstractly, it's like the model is dreaming of a game that it played a lot of, and real time inputs just change the state of the dream. It makes me wonder if hu…

Penrose (Nobel prize in physics) stipulates that quantum effects in the brain may allow a certain amount of time travel and back propagation to accomplish this.

You don't need back propagation to learn

This is an incredibly complex hypothesis that doesn't really seem justified by the evidence

Re: Diffusion models are real-time game engines

#277

Post by Workaccount2 »

jetrink wrote: post #261

What if instead of a video game, this was trained on video and control inputs from people operating equipment like warehouse robots? Then an automated system could visualize the result of a proposed action or series of actions when operating the equipment itself. You would need a different model/algorithm to propose control inputs, but this would offer a way for the system to validate and refine plans as part of a pr…

>Robotic Transformer 2 (RT-2) is a novel vision-language-action (VLA) model that learns from both web and robotics data, and translates this knowledge into generalised instructions for robotic control

https://deepmind.google/discover/blog/rt-2-new-model-transla...

Re: Diffusion models are real-time game engines

#278

Post by wrsh07 »

wkcheng wrote: post #9

It's insane that that this works, and that it works fast enough to render at 20 fps. It seems like they almost made a cross between a diffusion model and an RNN, since they had to encode the previous frames and actions and feed it into the model at each step. Abstractly, it's like the model is dreaming of a game that it played a lot of, and real time inputs just change the state of the dream. It makes me wonder if hu…

Makes me wonder when an update to the world models paper comes out where they drop in diffusion models: https://worldmodels.github.io/

Re: Diffusion models are real-time game engines

#279

Post by wavemode »

TeMPOraL wrote: post #263
wavemode wrote: post #234

Earlier quoted context omitted.

Yes it definitely is very good for simulating gameplay footage, don't get me wrong. Its input for predicting the next frame is not just the previous frame, it has access to a whole sequence of prior frames. But to say the model is simulating actual gameplay (i.e. that a person could actually play Doom in this) is far fetched. It's definitely great that the model was able to remember that the gray wall was still there…

> it's untenable for actual gameplay that the wall completely changed location and orientation. It would in an SCP-themed game. Or dreamscape/Inception themed one. Hell, "you're trapped in Doom-like dreamscape, escape before you lose your mind" is a very interesting pitch for a game. Basically take this Doom thing and make walking though a specific, unique-looking doorway from the original game to be the victory cond…

To be honest, I agree! That would be an interesting gameplay concept for sure.

Mainly just wanted to temper expectations I'm seeing throughout this thread that the model is actually simulating Doom. I don't know what will be required to get from here to there, but we're definitely not there yet.

Re: Diffusion models are real-time game engines

#280

Post by robotresearcher »

dewarrn1 wrote: post #221
wavemode wrote: post #190

Earlier quoted context omitted.

> Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected To temper this a bit, you may want to pay close attention to the demo videos. The player rarely backtracks, and for good reason - the few times the character does turn around and look back at something a second time, it has changed significantly (the most noticeable I think is the room with the grey wall and triangle sign…

Great observation. And not entirely unlike normal human visual perception which is notoriously vulnerable to missing highly salient information; I'm reminded of the "gorillas in our midst" work by Dan Simons and Christopher Chabris [0]. [0]: https://en.wikipedia.org/wiki/Inattentional_blindness#Invisi...

Not noticing to a gorilla that ‘shouldn’t’ be there is not the same thing as object permanence. Even quite young babies are surprised by objects that go missing.
Post reply on HN