Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Diffusion models are real-time game engines

gamengen.github.io

Post reply on HN
Print view View original

361–370 of 430 posts

Re: Diffusion models are real-time game engines

#361

Post by throwthrowuknow »

Sohcahtoa82 wrote: post #341
ninetyninenine wrote: post #333

Earlier quoted context omitted.

I don’t think this is not useful. This is a stepping stone for generating entire novel games.

> This is a stepping stone for generating entire novel games. I don't see how. This game "engine" is purely mapping [pixels, input] -> new pixels. It has no notion of game state (so you can kill an enemy, turn your back, then turn around again, and the enemy could be alive again), not to mention that it requires the game to already exist in order to train it. I suppose, in theory, you could train the network to inclu…

Read the paper. It is capable of maintaining state for a fairly long time including updating the UI elements.

Re: Diffusion models are real-time game engines

#362

Post by throwthrowuknow »

dtagames wrote: post #224

A diffusion model cannot be a game engine because a game engine can be used to create new games and modify the rules of existing games in real time -- even rules which are not visible on-screen. These tools are fascinating but, as with all AI hype, they need a disclaimer: The tool didn't create the game. It simply generated frames and the appearance of play mechanics from a game it sampled (which humans created).

They only trained it on one game and only embedded the control inputs. You could train it on many games and embed a lot more information about each of them which could possibly allow you to specify a prompt that would describe the game and then play it.

Re: Diffusion models are real-time game engines

#363

Post by Kapura »

What is useful about this? I am a game programmer, and I cannot imagine a world where this improves any part of the development process. It seems to me to be a way to copy a game without literally copying the assets and code; plagiarism with extra steps. What am I missing?

Re: Diffusion models are real-time game engines

#364

Post by meheleventyone »

GaggiX wrote: post #306
meheleventyone wrote: post #299

Earlier quoted context omitted.

No, I think I've been pretty clear that I'm interested in how mechanically sound the simulation is. Also those measures are over an even shorter duration so even less relevant to how coherent it is at real game scales.

How should this be concretely evaluated and measured? A vibe check?

I think the studies evaluation using very short video and humans is much more of a vibe check than what I’ve suggested.

Off the top of my head DOOM is open source so it should be reasonable to setup repeatable scenarios and use some frames from the game to create a starting scenario for the simulation that is the same. Then the input from the player of the game could be used to drive the simulated version. You could go further and instrument events occurring in the game for direct comparison to the simulation. I’d be interested in setting a baseline for playtime of the level in question and using sessions of around that length as an ultimate test.

There are some on obvious mechanical deficiencies seen in the videos they’ve published. One that really stood out to me was the damage taken when in the radioactive slime. So I don’t think the analysis would need to particularly deep to find differences.

Re: Diffusion models are real-time game engines

#365

Post by slashdave »

nopakos wrote: post #82
SeanAnderson wrote: post #64

Earlier quoted context omitted.

I think you're mistaken. The abstract says it's interactive, "We present GameNGen, the first game engine powered entirely by a neural model that enables real-time interaction" Further - "a diffusion model is trained to produce the next frame, conditioned on the sequence of past frames and actions." specifically "and actions" User input is being fed into this system and subsequent frames take that into account. The us…

Maybe it's so advanced, it knows the players' next moves, so it is a video!

I guess you are being sarcastic, except this is precisely what it is doing. And it's not hard: player movement is low information and probably not the hardest part of the model.

Re: Diffusion models are real-time game engines

#366

Post by ploxiln »

Sohcahtoa82 wrote: post #329

It's always fun reading the dead comments on a post like this. People love to point how how pointless this is. Some of ya'll need to learn how to make things for the fun of making things . Is this useful? No, not really. Is it interesting? Absolutely. Not everything has to be made for profit. Not everything has to be made to make the world a better place. Sometimes, people create things just for the learning experien…

The skepticism and criticism in this thread is against the hype of AI, it's implied by people saying "this is so amazing" that they think that in some near future you can create any video game experience you can imagine by just replacing all the software with some AI models, rendering the whole game.

When in reality this is the least efficient and reliable form of Doom yet created, using literally millions of times the computation used by the first x86 PCs that were able to render and play doom in real-time.

But it's a funny party trick, sure.

Re: Diffusion models are real-time game engines

#367

Post by sharpshadow »

TeMPOraL wrote: post #325
sharpshadow wrote: post #259

Earlier quoted context omitted.

But is it playing the actual game or just making a interactive video of it?

Yes. All video games are, by definition, interactive videos. What I imagine you're asking about is, a typical game like Doom is effectively a function: f(internal state, player input) -> (new frame, new internal state) where internal state is the shape and looks of loaded map, positions and behaviors and stats of enemies, player, items, etc. A typical AI that plays Doom, which is not what's happening here, is (at run…

Thank you for the further explanation, that’s what I thought in the meantime and intended to find out with my question.

That opens up a new branch of possibilities.

Re: Diffusion models are real-time game engines

#368

Post by dewarrn1 »

SeanAnderson wrote: post #314

After some discussion in this thread, I found it worth pointing out that this paper is NOT describing a system which receives real-time user input and adjusts its output accordingly, but, to me, the way the abstract is worded heavily implied this was occurring. It's trained on a large set of data in which agents played DOOM and video samples are given to users for evaluation, but users are not feeding inputs into the…

The paper should definitely be more clear on this point, but there's a sentence in section 5.2.3 that makes me think that this was playable and played: "When playing with the model manually, we observe that some areas are very easy for both, some areas are very hard for both, and in some the agent performs much better." It may be a failure of imagination, but I can't think of another reasonable way of interpreting "playing with the model manually".

Re: Diffusion models are real-time game engines

#369

Post by sitkack »

Jonovono wrote: post #357
sitkack wrote: post #55

What most programmers don't understand, that in the very near future, the entire application will be delivered by an AI model, no source, no text, just connect to the app over RDP. The whole app will be created by example, the app developer will train the app like a dog trainer trains a dog.

I think it's possible AI models will generate dynamic UI for each client and stream the UI to clients (maybe eventually client devices will generate their UI on the fly) similar to Google Stadia. Maybe some offset of video that allows the remote to control it. Maybe Wasm based - just stream wasm bytecode around? The guy behind VLC is building a library for ulta low latency: https://www.kyber.video/techology . I was p…

I am so glad you posted, this is super cool!

I too have been thinking about how to push dynamic wasm to the client for super low latency UIs.

LiveView is just the beginning. Your readme is dreamy. I'll dive into your project at the end of Sept when I get back into deep tech.

Re: Diffusion models are real-time game engines

#370

Post by Chance-Device »

foota wrote: post #354
Chance-Device wrote: post #331

Earlier quoted context omitted.

I also thought this, but refer back to the paper, not the abstract: > A is the set of key presses and mouse movements… > …to condition on actions, we simply learn an embedding A_emb for each action So, it’s clear that in this model the diffusion process is conditioned by embedding A that is derived from user actions rather than words. Then a noised start frame is encoded into latents and concatenated on to the noise…

I wonder if they could somehow feed in a trained Gaussian splats model to this to get better images? Since the splats are specifically designed for rendering it seems like it would be an efficient way for the image model to learn the geometry without having to encode it on the image model itself.

I’m not sure how that would help vs just training the model with the conditionings described in the paper.

I’m not very familiar with Gaussian splats models, but aren’t they just a way of constructing images using multiple superimposed parameterized Gaussian distributions, sort of like the Fourier series does with waveforms using sine and cosine waves?

I’m not seeing how that would apply here but I’d be interested in hearing how you would do it.

Post reply on HN