Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Diffusion models are real-time game engines

gamengen.github.io

Post reply on HN
Print view View original

341–350 of 430 posts

Re: Diffusion models are real-time game engines

#341

Post by Sohcahtoa82 »

ninetyninenine wrote: post #333
Sohcahtoa82 wrote: post #329

It's always fun reading the dead comments on a post like this. People love to point how how pointless this is. Some of ya'll need to learn how to make things for the fun of making things . Is this useful? No, not really. Is it interesting? Absolutely. Not everything has to be made for profit. Not everything has to be made to make the world a better place. Sometimes, people create things just for the learning experien…

I don’t think this is not useful. This is a stepping stone for generating entire novel games.

> This is a stepping stone for generating entire novel games.

I don't see how.

This game "engine" is purely mapping [pixels, input] -> new pixels. It has no notion of game state (so you can kill an enemy, turn your back, then turn around again, and the enemy could be alive again), not to mention that it requires the game to already exist in order to train it.

I suppose, in theory, you could train the network to include game state in the input and output, or potentially even handle game state outside the network entirely and just make it one of the inputs, but the output would be incredibly noisy and nigh unplayable.

And like I said, all of it requires the game to already exist in order to train the network.

Re: Diffusion models are real-time game engines

#342

Post by lewhoo »

SeanAnderson wrote: post #314

After some discussion in this thread, I found it worth pointing out that this paper is NOT describing a system which receives real-time user input and adjusts its output accordingly, but, to me, the way the abstract is worded heavily implied this was occurring. It's trained on a large set of data in which agents played DOOM and video samples are given to users for evaluation, but users are not feeding inputs into the…

The movement of the player seems jittery a bit so I inferred something similar on that basis.

Re: Diffusion models are real-time game engines

#343

Post by airstrike »

Sohcahtoa82 wrote: post #341
ninetyninenine wrote: post #333

Earlier quoted context omitted.

I don’t think this is not useful. This is a stepping stone for generating entire novel games.

> This is a stepping stone for generating entire novel games. I don't see how. This game "engine" is purely mapping [pixels, input] -> new pixels. It has no notion of game state (so you can kill an enemy, turn your back, then turn around again, and the enemy could be alive again), not to mention that it requires the game to already exist in order to train it. I suppose, in theory, you could train the network to inclu…

> (so you can kill an enemy, turn your back, then turn around again, and the enemy could be alive again)

Sounds like a great game.

> not to mention that it requires the game to already exist in order to train it

Diffusion models create new images that did not previously exist all of the time, so I'm not sure how that follows. It's not hard to extrapolate from TFA to a model that generically creates games based on some input

Re: Diffusion models are real-time game engines

#344

Post by ninetyninenine »

Sohcahtoa82 wrote: post #341
ninetyninenine wrote: post #333

Earlier quoted context omitted.

I don’t think this is not useful. This is a stepping stone for generating entire novel games.

> This is a stepping stone for generating entire novel games. I don't see how. This game "engine" is purely mapping [pixels, input] -> new pixels. It has no notion of game state (so you can kill an enemy, turn your back, then turn around again, and the enemy could be alive again), not to mention that it requires the game to already exist in order to train it. I suppose, in theory, you could train the network to inclu…

>It has no notion of game state (so you can kill an enemy, turn your back, then turn around again)

Well you see a wall you turn around then turn back the wall is still there. With enough training data the model will be able to pick up the state of the enemy because it has ALREADY learned the state of the wall due to much more numerous data on the wall. It's probably impractical to do this, but this is only a stepping stone like said.

> not to mention that it requires the game to already exist in order to train it.

Is this a problem? Do games not exist? Not only due we have tons of games, but we also have in theory unlimited amounts of training data for each game.

Re: Diffusion models are real-time game engines

#345

Post by KhoomeiK »

NVIDIA did something similar with GANs in 2020 [1], except users could actually play those games (unlike in this diffusion work which just plays back simulated video). Sentdex later adapted this to play GTA with a really cool demo [2].

[1] https://research.nvidia.com/labs/toronto-ai/gameGAN/

[2] https://www.youtube.com/watch?v=udPY5rQVoW0

Re: Diffusion models are real-time game engines

#346

Post by nielsbot »

wavemode wrote: post #190
vessenes wrote: post #2

So, this is surprising. Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected, which would be roughly ‘none’. Google here uses SD 1.4, as the core of the diffusion model, which is a nice reminder that open models are useful to even giant cloud monopolies. The two main things of note I took away from the summary were: 1) they got infinite training data using agents playing doom…

> Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected To temper this a bit, you may want to pay close attention to the demo videos. The player rarely backtracks, and for good reason - the few times the character does turn around and look back at something a second time, it has changed significantly (the most noticeable I think is the room with the grey wall and triangle sign…

You can also notice in the first part of the video the ammo numbers fluctuate a bit randomly.

Re: Diffusion models are real-time game engines

#347

Post by godelski »

tobr wrote: post #142
snickmy wrote: post #129

Earlier quoted context omitted.

Those are valid points, but irrelevant for the context of this research. Yes, the computational cost is ridicolous compared to the original game, and yes, it lacks basic things like pre-computing, storing, etc. That said, you could assume that all that can be either done at the margin of this discovery OR over time will naturally improve OR will become less important as a blocker. The fact that you can model a sequen…

I suppose it also doesn't really matter what kinds of resources the game originally requires. The diffusion model isn't going to require twice as much memory just because the game does. Presumably you wouldn't even necessarily need to be able to render the original game in real time - I would imagine the basic technique would work even if you used a state of the Hollywood-quality offline renderer to render each input…

Well the majority of ML systems are compression machines (entropy minimizers), so ideally you'd want to see if you can learn the assets and game mechanics through play alone (what this paper shows). Better would be to do so more efficiently than that devs themselves, finding better compression. Certainly the game is not perfectly optimized. But still, this is a step in that direction. I mean no one has accomplished this before so even with a model with far higher capacity it's progress. (I think people are interpreting my comment as dismissive. I'm critiquing but the key point I was making was about how there's likely better architectures, training methods, and all sorts of stuff to still research. Personally I'm glad there's still more to research. That's the fun part)

Re: Diffusion models are real-time game engines

#348

Post by sitkack »

Grimblewald wrote: post #70
sitkack wrote: post #55

What most programmers don't understand, that in the very near future, the entire application will be delivered by an AI model, no source, no text, just connect to the app over RDP. The whole app will be created by example, the app developer will train the app like a dog trainer trains a dog.

that might work for some applications, especially recreational things, I think we're a while away from it doing away with all things, especially where deterministic behavior, efficiency, or reliability are important.

Problems for two papers down the line.

Re: Diffusion models are real-time game engines

#349

Post by Sohcahtoa82 »

ninetyninenine wrote: post #344
Sohcahtoa82 wrote: post #341

Earlier quoted context omitted.

> This is a stepping stone for generating entire novel games. I don't see how. This game "engine" is purely mapping [pixels, input] -> new pixels. It has no notion of game state (so you can kill an enemy, turn your back, then turn around again, and the enemy could be alive again), not to mention that it requires the game to already exist in order to train it. I suppose, in theory, you could train the network to inclu…

>It has no notion of game state (so you can kill an enemy, turn your back, then turn around again) Well you see a wall you turn around then turn back the wall is still there. With enough training data the model will be able to pick up the state of the enemy because it has ALREADY learned the state of the wall due to much more numerous data on the wall. It's probably impractical to do this, but this is only a stepping…

> Well you see a wall you turn around then turn back the wall is still there. With enough training data the model will be able to pick up the state of the enemy because it has ALREADY learned the state of the wall due to much more numerous data on the wall.

It's really important to understand that ALL THE MODEL KNOWS is a mapping of [pixels, input] -> new pixels. It has zero knowledge of game state. The wall is still there after spinning 360 degrees simply because it knows that the image of a view facing away from the wall while holding the key to turn right eventually becomes an image of a view of the wall.

The only "state" that is known is the last few frames of the game screen. Because of this, it's simply not possible for the game model to know if an enemy should be shown as dead or alive once it has been off-screen for longer than those few frames. It also means that if you keeping turning away and towards an enemy, it could teleport around. Once it's off the screen for those few frames, the model will have forgotten about it.

> Is this a problem? Do games not exist?

If you're trying to make a new game, then you need new frames to train the model on.

Re: Diffusion models are real-time game engines

#350

Post by darby_nine »

stnmtn wrote: post #305
darby_nine wrote: post #238

Earlier quoted context omitted.

> Simply?! I understand it's mechanically trivial but the fact that it's compressed such a rich conditional distribution seems far from simple to me. It's much simpler than actually creating a game....

If someone told you 10 years ago that they were going to create something where you could play a whole new level of Doom, without them writing a single line of game logic/rendering code, would you say that that is simpler than creating a demo by writing the game themselves?

There are two things at play here: the complexity of the underlying mechanism, and the complexity of detailed creation. This is obviously a complicated mechanism, but in another sense it's a trivial result compared to actually reproducing the game itself in its original intended state.
Post reply on HN