Live data from Hacker News

Diffusion models are real-time game engines

gamengen.github.io

321–330 of 430 posts

Re: Diffusion models are real-time game engines

#321

Earlier quoted context omitted.

To be honest, I agree! That would be an interesting gameplay concept for sure. Mainly just wanted to temper expectations I'm seeing throughout this thread that the model is actually simulating Doom. I don't know what will be required to get from here to there, but we're definitely not there yet.

What you're pointing at mirrors the same kind of limitation in using LLMs for role-play/interactive fictions.

Maybe a hybrid approach would work. Certain things like inventory being stored as variables, lists etc.

Wouldn't be as pure though.

Re: Diffusion models are real-time game engines

#322

After some discussion in this thread, I found it worth pointing out that this paper is NOT describing a system which receives real-time user input and adjusts its output accordingly, but, to me, the way the abstract is worded heavily implied this was occurring. It's trained on a large set of data in which agents played DOOM and video samples are given to users for evaluation, but users are not feeding inputs into the…

You are incorrect, this is an interactive simulation that is playable by humans. > Figure 1: a human player is playing DOOM on GameNGen at 20 FPS. The abstract is ambiguously worded which has caused a lot of confusion here, but the paper is unmistakably clear about this point. Kind of disappointing to see this misinformation upvoted so highly on a forum full of tech experts.

Yeah. If isn't doing this, then what could it be doing that is worth a paper? "real-time user input and adjusts its output accordingly"

Re: Diffusion models are real-time game engines

#323

Has this model actually learned the 3d space of the game? Is it possible to break the camera free and roam around the map freely and view it from different angles? I noticed a few hallucinations e.g. when it picked green jacket from a corner, walking back it generated another corner. Therefore I don't think it has any clue about the 3D world of the game at all.

> Therefore I don't think it has any clue about the 3D world of the game at all.

AI models don't "know" things at all.

At best, they're just very fuzzy predictors. In this case, given the last couple frames of video and a user input, it predicts the next frame.

It has zero knowledge of the game world, game rules, interactions, etc. It's merely a mapping of [pixels, input] -> pixels.

Re: Diffusion models are real-time game engines

#324

Earlier quoted context omitted.

Where does a sora video turn around backwards? I can’t maintain such consistency in my own dreams.

I don't know of an example (not to say it doesn't exist) but the problem is fundamentally the same as things moving out of sight/out of frame and coming back again.

> the problem is fundamentally the same as things moving out of sight/out of frame and coming back again

Maybe it is, but doing that with the entire scene instead of just a small part of it makes the problem massively harder, as the model needs to grow exponentially to remember more things. It isn't something that we will manage anytime soon, maybe 10-20 years with current architecture and same compute progress.

Then you make that even harder by remembering a whole game level? No, ain't gonna happen in our lifetimes without massive changes to the architecture. They would need to make a different model keep track of level state etc, not just an image to image model.

Re: Diffusion models are real-time game engines

#325

Earlier quoted context omitted.

No, it implements a 3D FPS that's interactive, and renders each frame based on your input and a lot of memorized gameplay.

But is it playing the actual game or just making a interactive video of it?

Yes.

All video games are, by definition, interactive videos.

What I imagine you're asking about is, a typical game like Doom is effectively a function:

  f(internal state, player input) -> (new frame, new internal state)
where internal state is the shape and looks of loaded map, positions and behaviors and stats of enemies, player, items, etc.

A typical AI that plays Doom, which is not what's happening here, is (at runtime):

  f(last frame) -> new player input
and is attached in a loop to the previous case in the obvious way.

What we have here, however, is a game you can play but implemented in a diffusion model, and it works like this:

  f(player input, N last frames) -> new frame
Of note here is the lack of game state - the state is implicit in the contents of the N previous frames, and is otherwise not represented or mutated explicitly. The diffusion model has seen so much Doom that it, in a way, internalized most of the state and its evolution, so it can look at what's going on and guess what's about to happen. Which is what it does: it renders the next frame by predicting it, based on current user input and last N frames. And then that frame becomes the input for the next prediction, and so on, and so on.

So yes, it's totally an interactive video and a game and a third thing - a probabilistic emulation of Doom on a generative ML model.

Re: Diffusion models are real-time game engines

#326

After some discussion in this thread, I found it worth pointing out that this paper is NOT describing a system which receives real-time user input and adjusts its output accordingly, but, to me, the way the abstract is worded heavily implied this was occurring. It's trained on a large set of data in which agents played DOOM and video samples are given to users for evaluation, but users are not feeding inputs into the…

I knew it was too good be true but seems like real time video generation can be good enough to get to a point where it feels like a truly interactive video/game

Imagine if text2game was possible. there would be some sort of network generating each frame from an image generated by text, with some underlying 3d physics simulation to keep all the multiplayer screens sync'd

this paper does not seem to be of that possibility rather some cleverly words to make you think people were playing a real time video. we can't even generate more than 5~10 second of video without it hallucinating. something this persistent would require an extreme amount of gameplay video training. it can be done but the video shown by this paper is not true to its words.

Re: Diffusion models are real-time game engines

#327

Earlier quoted context omitted.

You are incorrect, this is an interactive simulation that is playable by humans. > Figure 1: a human player is playing DOOM on GameNGen at 20 FPS. The abstract is ambiguously worded which has caused a lot of confusion here, but the paper is unmistakably clear about this point. Kind of disappointing to see this misinformation upvoted so highly on a forum full of tech experts.

Yeah. If isn't doing this, then what could it be doing that is worth a paper? "real-time user input and adjusts its output accordingly"

There is a hint in the paper itself:

It says in a shy way that it is based on: "Ha & Schmidhuber (2018) who train a Variational Auto-Encoder (Kingma & Welling, 2014) to encode game frames into a latent vector"

So it means they most likely took https://worldmodels.github.io/ (that is actually open-source) or something similar and swapped the frame generation by Stable Diffusion that was released in 2022.

Re: Diffusion models are real-time game engines

#328

Earlier quoted context omitted.

> Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected To temper this a bit, you may want to pay close attention to the demo videos. The player rarely backtracks, and for good reason - the few times the character does turn around and look back at something a second time, it has changed significantly (the most noticeable I think is the room with the grey wall and triangle sign…

Great observation. And not entirely unlike normal human visual perception which is notoriously vulnerable to missing highly salient information; I'm reminded of the "gorillas in our midst" work by Dan Simons and Christopher Chabris [0]. [0]: https://en.wikipedia.org/wiki/Inattentional_blindness#Invisi...

Work which exaggerates the blindness.

The people were told to focus very deeply on a certain aspect of the scene. Maintaining that focus means explicitly blocking things not related to that focus. Also, there is social pressure at the end to have peformed well at the task; evaluating them on a task which is intentionally completely different than the one explicitly given is going to bias people away from reporting gorillas.

And also, "notice anything unusual" is a pretty vague prompt. No-one in the video thought the gorillas were unusual, so if the PEOPLE IN THE SCENE thought gorillas were normal, why would I think they were strange? Look at any TV show, they are all full of things which are pretty crazy unusual in normal life, yet not unusual in terms of the plot.

Why would you think the gorillas were unusual?

Re: Diffusion models are real-time game engines

#329
It's always fun reading the dead comments on a post like this. People love to point how how pointless this is.

Some of ya'll need to learn how to make things for the fun of making things. Is this useful? No, not really. Is it interesting? Absolutely.

Not everything has to be made for profit. Not everything has to be made to make the world a better place. Sometimes, people create things just for the learning experience, the challenge, or they're curious to see if something is possible.

Time spent enjoying yourself is never time wasted. Some of ya'll are going to be on your death beds wishing you had allowed yourself to have more fun.

Re: Diffusion models are real-time game engines

#330

It's always fun reading the dead comments on a post like this. People love to point how how pointless this is. Some of ya'll need to learn how to make things for the fun of making things . Is this useful? No, not really. Is it interesting? Absolutely. Not everything has to be made for profit. Not everything has to be made to make the world a better place. Sometimes, people create things just for the learning experien…

So true. The hustle culture is an spreading disease that has replaced the fun maker culture from the 80s/90s.

It's unavoidable though. Cost of living being increasingly expensive and romantization of entrepreneurs like they are rock stars leads towards this hustle mindset.

Post reply on HN