Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Diffusion models are real-time game engines

gamengen.github.io

Post reply on HN
Print view View original

331–340 of 430 posts

Re: Diffusion models are real-time game engines

#331

Post by Chance-Device »

SeanAnderson wrote: post #314

After some discussion in this thread, I found it worth pointing out that this paper is NOT describing a system which receives real-time user input and adjusts its output accordingly, but, to me, the way the abstract is worded heavily implied this was occurring. It's trained on a large set of data in which agents played DOOM and video samples are given to users for evaluation, but users are not feeding inputs into the…

I also thought this, but refer back to the paper, not the abstract:

> A is the set of key presses and mouse movements…

> …to condition on actions, we simply learn an embedding A_emb for each action

So, it’s clear that in this model the diffusion process is conditioned by embedding A that is derived from user actions rather than words.

Then a noised start frame is encoded into latents and concatenated on to the noise latents as a second conditioning.

So we have a diffusion model which is trained solely on images of doom, and which is conditioned on current doom frames and user actions to produce subsequent frames.

So yes, the users are playing it.

However, it should be unsurprising that this is possible. This is effectively just a neural recording of the game. But it’s a cool tech demo.

Re: Diffusion models are real-time game engines

#332

Post by godelski »

_hark wrote: post #177
godelski wrote: post #130

Earlier quoted context omitted.

> With enough computation, your neural net weights would converge to some very compressed latent representation of the source code of DOOM. You and I have very different definitions of compression https://news.ycombinator.com/item?id=41377398 > Someone in the field could probably correct me on that. ^__^

The raw capacity of the network doesn't tell you how complex the weights actually are. The capacity is only an upper bound on the complexity. It's easy to see this by noting that you can often prune networks quite a bit without any loss in performance. I.e. the effective dimension of the manifold the weights live on can be much, much smaller than the total capacity allows for. In fact, good regularization is exactly…

I think your confusing capacity with the training dynamics.

Capacity is autological. The amount of information it can express.

Training dynamics are the way the model learns, the optimization process, etc. So this is where things like regularization come into play.

There's also architecture which affects the training dynamics as well as model capacity. Which makes no guarantee that you get the most information dense representation.

Fwiw, the authors did also try distillation.

Re: Diffusion models are real-time game engines

#333

Post by ninetyninenine »

Sohcahtoa82 wrote: post #329

It's always fun reading the dead comments on a post like this. People love to point how how pointless this is. Some of ya'll need to learn how to make things for the fun of making things . Is this useful? No, not really. Is it interesting? Absolutely. Not everything has to be made for profit. Not everything has to be made to make the world a better place. Sometimes, people create things just for the learning experien…

I don’t think this is not useful. This is a stepping stone for generating entire novel games.

Re: Diffusion models are real-time game engines

#334

Post by msk-lywenn »

Sohcahtoa82 wrote: post #329

It's always fun reading the dead comments on a post like this. People love to point how how pointless this is. Some of ya'll need to learn how to make things for the fun of making things . Is this useful? No, not really. Is it interesting? Absolutely. Not everything has to be made for profit. Not everything has to be made to make the world a better place. Sometimes, people create things just for the learning experien…

I’d like to now to carbon footprint of that fun.

Re: Diffusion models are real-time game engines

#335

Post by momojo »

dtagames wrote: post #224

A diffusion model cannot be a game engine because a game engine can be used to create new games and modify the rules of existing games in real time -- even rules which are not visible on-screen. These tools are fascinating but, as with all AI hype, they need a disclaimer: The tool didn't create the game. It simply generated frames and the appearance of play mechanics from a game it sampled (which humans created).

The title should be "Diffusion Models can be used to render frames given user input"

Re: Diffusion models are real-time game engines

#336

Post by teamonkey »

SeanAnderson wrote: post #314

After some discussion in this thread, I found it worth pointing out that this paper is NOT describing a system which receives real-time user input and adjusts its output accordingly, but, to me, the way the abstract is worded heavily implied this was occurring. It's trained on a large set of data in which agents played DOOM and video samples are given to users for evaluation, but users are not feeding inputs into the…

I think someone is playing it, but it has a reduced set of inputs and they're playing it in a very specific way (slowly, avoiding looking back to places they've been) so as not to show off the flaws in the system.

The people surveyed in this study are not playing the game, they are watching extremely short video clips of the game being played and comparing them to equally short videos of the original Doom being played, to see if they can spot the difference.

I may be wrong with how it works, but I think this is just hallucinating in real time. It has no internal state per se, it knows what was on screen in the previous few frames and it knows what inputs the user is pressing, and so it generates the next frame. Like with video compression, it probably doesn't need to generate a full frame every time, just "differences".

As with all the previous AI game research, these are not games in any real sense. They fall apart when played beyond any meaningful length of time (seconds). Crucially, they are not playable by anyone other than the developers in very controlled settings. A defining attribute of any game is that it can be played.

Re: Diffusion models are real-time game engines

#337

Post by slashdave »

SeanAnderson wrote: post #313
slashdave wrote: post #310

Earlier quoted context omitted.

No, I am not. The interaction is part of the training, and is used during inference, but it is not including during the process of generation.

Okay, I think you're right. My mistake. I read through the paper more closely and I found the abstract to be a bit misleading compared to the contents. Sorry.

Don't worry. The paper is not very well written.

Re: Diffusion models are real-time game engines

#338

Post by dewarrn1 »

throwway_278314 wrote: post #328
dewarrn1 wrote: post #221

Earlier quoted context omitted.

Great observation. And not entirely unlike normal human visual perception which is notoriously vulnerable to missing highly salient information; I'm reminded of the "gorillas in our midst" work by Dan Simons and Christopher Chabris [0]. [0]: https://en.wikipedia.org/wiki/Inattentional_blindness#Invisi...

Work which exaggerates the blindness. The people were told to focus very deeply on a certain aspect of the scene. Maintaining that focus means explicitly blocking things not related to that focus. Also, there is social pressure at the end to have peformed well at the task; evaluating them on a task which is intentionally completely different than the one explicitly given is going to bias people away from reporting go…

I understand what you mean. I believe that the authors would contend that what you're describing is a typical attentional state for an awake/aware human: focused mostly on one thing, and with surprisingly little awareness of most other things (until/unless they are in turn attended).

Furthermore, even what we attend to isn't always represented with all that much detail. Simons has a whole series of cool demonstration experiments where they show that they can swap out someone you're speaking with (an unfamiliar conversational partner like a store clerk or someone asking for directions), and you may not even notice [0]. It's rather eerie.

[0]: https://www.youtube.com/watch?v=FWSxSQsspiQ&t=5s

Re: Diffusion models are real-time game engines

#339

Post by jsheard »

kridsdale1 wrote: post #307
TeMPOraL wrote: post #263

Earlier quoted context omitted.

> it's untenable for actual gameplay that the wall completely changed location and orientation. It would in an SCP-themed game. Or dreamscape/Inception themed one. Hell, "you're trapped in Doom-like dreamscape, escape before you lose your mind" is a very interesting pitch for a game. Basically take this Doom thing and make walking though a specific, unique-looking doorway from the original game to be the victory cond…

Check out the actual modern DOOM WAD MyHouse which implements these ideas. It totally breaks our preconceptions of what the DOOM engine is capable of. https://en.wikipedia.org/wiki/MyHouse.wad

MyHouse is excellent, but it mostly breaks our perception of what the Doom engine is capable of by not really using the Doom engine. It leans heavily on engine features which were embellishments by the GZDoom project, and never existed in the original Doom codebase.

Re: Diffusion models are real-time game engines

#340

Post by godelski »

snickmy wrote: post #160
godelski wrote: post #137

Earlier quoted context omitted.

I'm not sure what you're saying is irrelevant. 1) the model has enough memory to store not only all game assets and engine but even hundreds of "plays". 2) me mentioning that there's still a lot of room to make these things better (seems you think so too so maybe not this one?) 3) an interesting point I was wondering to compare current state of things (I mean I'll give you this but it's just a random thought and I'm…

Sorry, probably didn't explain myself well enough 1) yes you are correct. the point i was making is that, in the context of the discovery/research, that's outside the scope, and 'easier' to do, as it has been done in other verticals (ie.: e2e self driving) 2) yep, aligned here 3) I'm not fully following here, but agree this is not NeurIPS, and no Schmidhuber's bickering. 4) The network does store information, it just…

1) I'm not sure this is outside scope. It's also not something I'd use to reject a paper were I to review this in a conference. I mean you got to start somewhere and unlike reviewer 2 I don't think any criticism is rejection criteria. That'd be silly since lack of globally optimal solutions. But I'm also unconvinced this is proven my self-driving vehicles but I'm also not an RL expert.

3) It's always hard to evaluate. I was thinking about the ripping the game and so a reasonable metric is a comparison of ability to perform the task by a human. Of course I'm A LOT faster than my dishwasher at cleaning dishes but I'm not occupied while it is going, so it still has high utility. (Someone tell reviewer 2 lol)

4) Why should we believe that it doesn't store gameplay? The model was fed "user" inputs and frames. So it has this information and this information appears useful for learning the task.

Post reply on HN