Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Diffusion models are real-time game engines

gamengen.github.io

Post reply on HN
Print view View original

171–180 of 430 posts

Re: Diffusion models are real-time game engines

#171

Post by lokimedes »

dbspin wrote: post #127
lokimedes wrote: post #33

Earlier quoted context omitted.

It makes good sense for humans to have this ability. If we flip the argument, and see the next frame as a hypothesis for what is expected as the outcome of the current frame, then comparing this "hypothesis" with what is sensed makes it easier to process the differences, rather than the totality of the sensory input. As Richard Dawkins recently put it in a podcast[1], our genes are great prediction machines, as their…

Worth noting that aphantasia doesn't necessarily extend to dreams. Anecdotally - I have pretty severe aphantasia (I can conjure milisecond glimpses of barely tangible imagery that I can't quite perceive before it's gone - but only since learning that visualisation wasn't a linguistic metaphor). I can't really simulate object rotation. I can't really 'picture' how things will look before they're drawn / built etc. How…

I fabulate about this in another comment below:

> Many people with aphantasia reports being able to visualize in their dreams, meaning that they don't lack the ability to generate visuals. So it may be that the [aphantasia] brain has an affinity to rely on the abstract representation when "thinking", while dreaming still uses the "stable diffusion mode".

(I obviously don't know what I'm talking about, just a fellow aphant)

Re: Diffusion models are real-time game engines

#172

Post by kqr »

I have been kind of "meh" about the recent AI hype, but this is seriously impressive.

Of course, we're clearly looking at complete nonsense generated by something that does not understand what it is doing – yet, it is astonishingly sensible nonsense given the type of information it is working from. I had no idea the state of the art was capable of this.

Re: Diffusion models are real-time game engines

#174

Post by jsheard »

modeless wrote: post #36
danjl wrote: post #4

So, diffusion models are game engines as long as you already built the game? You need the game to train the model. Chicken. Egg?

If you train it on multiple games then you could produce new games that have never existed before, in the same way image generation models can produce new images that have never existed before.

It's unlikely that such a procedurally generated mashup would be perfectly coherent, stable and most importantly fun right out of the gate, so you would need some way to reach into the guts of the generated game and refine it. If properties as simple as "how much health this enemy type has" are scattered across an enormous inscrutable neural network, and may not even have a single consistent definition in all contexts, that's going to be quite a challenge. Nevermind if the game just catastrophically implodes and you have to "debug" the model.

Re: Diffusion models are real-time game engines

#176

Post by ccozan »

injidup wrote: post #68
kragen wrote: post #10

Earlier quoted context omitted.

here are some ideas: - you could build a non-real-time version of the game engine and use the neural net as a real-time approximation - you could edit videos shot in real life to have huds or whatever and train the neural net to simulate reality rather than doom. (this paper used 900 million frames which i think is about a year of video if it's 30fps, but maybe algorithmic improvements can cut the training requiremen…

Why games? I will train it on 1 years worth of me attending Microsoft teams meetings. Then I will go surfing.

most underrated comment here!

Re: Diffusion models are real-time game engines

#177

Post by _hark »

godelski wrote: post #130
mlsu wrote: post #31

Earlier quoted context omitted.

With enough computation, your neural net weights would converge to some very compressed latent representation of the source code of DOOM. Maybe smaller even than the source code itself? Someone in the field could probably correct me on that. At which point, you effectively would be interpolating in latent space through the source code to actually "render" the game. You'd have an entire latent space computer, with an…

> With enough computation, your neural net weights would converge to some very compressed latent representation of the source code of DOOM. You and I have very different definitions of compression https://news.ycombinator.com/item?id=41377398 > Someone in the field could probably correct me on that. ^__^

The raw capacity of the network doesn't tell you how complex the weights actually are. The capacity is only an upper bound on the complexity.

It's easy to see this by noting that you can often prune networks quite a bit without any loss in performance. I.e. the effective dimension of the manifold the weights live on can be much, much smaller than the total capacity allows for. In fact, good regularization is exactly that which encourages the model itself to be compressible.

Re: Diffusion models are real-time game engines

#178

Post by kqr »

smusamashah wrote: post #170

Has this model actually learned the 3d space of the game? Is it possible to break the camera free and roam around the map freely and view it from different angles? I noticed a few hallucinations e.g. when it picked green jacket from a corner, walking back it generated another corner. Therefore I don't think it has any clue about the 3D world of the game at all.

> Is it possible to break the camera free and roam around the map freely and view it from different angles?

I would assume only if the training data contained this type of imagery, which it did not. The training data (from what I understand) consisted only of input+video of actual gameplay, so that is what the model is trained to mimick.

This is like a dog that has been trained to form English words – what's impressive is not that it does it well, but that it does it at all.

Re: Diffusion models are real-time game engines

#179

Post by ccozan »

cubefox wrote: post #90
mo_42 wrote: post #63

An implementation of the game engine in the model itself is theoretically the most accurate solution for predicting the next frame. I'm wondering when people will apply this to other areas like the real world. Would it learn the game engine of the universe (ie physics)?

A popular theory in neuroscience is that this is what the brain does: https://slatestarcodex.com/2017/09/05/book-review-surfing-un... It's called predictive coding. By trying to predict sensory stimuli, the brain creates a simplified model of the world, including common sense physics. Yann LeCun says that this is a major key to AGI. Another one is effective planning. But while current predictive models (autoregressiv…

Howver, this is due how we actually digitize video. From a human point a view, looking in my room reduces the load to the _objects_ in the room and everyhing else is just noise ( like the color of the wall could be just a single item to remember, while otherwise in the digital world, it needs to remember all the pixels )

Re: Diffusion models are real-time game engines

#180

Post by kqr »

injidup wrote: post #68
kragen wrote: post #10

Earlier quoted context omitted.

here are some ideas: - you could build a non-real-time version of the game engine and use the neural net as a real-time approximation - you could edit videos shot in real life to have huds or whatever and train the neural net to simulate reality rather than doom. (this paper used 900 million frames which i think is about a year of video if it's 30fps, but maybe algorithmic improvements can cut the training requiremen…

Why games? I will train it on 1 years worth of me attending Microsoft teams meetings. Then I will go surfing.

Even if you spend 40 hours a week in video conferences, you'll have to work for over four years to get one years' worth of footage. Of course, by then the models will be even better and so you might actually have a chance of going surfing.

I guess I should start hoarding video of myself now.

Post reply on HN