Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Diffusion models are real-time game engines

gamengen.github.io

Post reply on HN
Print view View original

131–140 of 430 posts

Re: Diffusion models are real-time game engines

#131

Post by marci »

icoder wrote: post #119

This is impressive. But at the same time, it can't count. We see this every time, and I understand why it happens, but it is still intriguing. We are so close or in some ways even way beyond, and yet at the same time so extremely far away, from 'our' intelligence. (I say it can't count because there are numerous examples where the bullet count glitches, it goes right impressively often, but still, counting, being up…

'our' intelligence may not be the best thing we can make. It would be like trying to only make planes that flaps wings or trucks with legs. A bit like using a llm to do multiplication. Not the best tool. Biomimcry is great for inspiration, but shouldn't be a 1-to-1 copy, especialy in different scale and medium.

Re: Diffusion models are real-time game engines

#133

Post by JDEngi »

This is going to be the future of cloud gaming, isn't it? In order to deal with the latency, we just generate the next frame locally, and we'll have the true frame coming in later from the cloud, so we're never dreaming too far ahead of the actual game.

Re: Diffusion models are real-time game engines

#134

Post by nine_k »

vessenes wrote: post #2

So, this is surprising. Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected, which would be roughly ‘none’. Google here uses SD 1.4, as the core of the diffusion model, which is a nice reminder that open models are useful to even giant cloud monopolies. The two main things of note I took away from the summary were: 1) they got infinite training data using agents playing doom…

But it's not a game. It's a memory of a game video, predicting the next frame based on the few previous frames, like "I can imagine what happened next".

I would call it the world's least efficient video compression.

What I would like to see is the actual predictive strength, aka imagination, which I did not notice mentioned in the abstract. The model is trained on a set of classic maps. What would it do, given a few frames of gameplay on an unfamiliar map as input? How well could it imagine what happens next?

Re: Diffusion models are real-time game engines

#135

Post by amelius »

THBC wrote: post #16
w_for_wumbo wrote: post #13

Earlier quoted context omitted.

That feels like the endgame of video game generation. You select an art style, a video and the type of game you'd like to play. The game is then generated in real-time responding to each action with respect to the existing rule engine. I imagine a game like that could get so convincing in its details and immersiveness that one could forget they're playing a game.

Holodeck is just around the corner

Except for haptics.

Re: Diffusion models are real-time game engines

#136

Post by MasterScrat »

HellDunkel wrote: post #118

Although impressive i must disagree. Diffusion models are not game engines. A game engine is a component to propell your game (along the time axis?). In that sense it is similar to the engine of the car, hence the name. It does not need a single working car nor a road to drive on do its job. The above is a dynamic, interactive replication of what happens when you put a car on a given road, requiring a million test dr…

Interesting point.

In a way this is a "simulated game engine", trained from actual game engine data. But I would argue a working simulated game engine becomes a game engine of its own, as it is then able to "propell the game" as you say. The way it achieves this becomes irrelevant, in one case the content was crafted by humans, in the other case it mimics existing game content, the player really doesn't care!

> An engine would also work offroad.

Here you could imagine that such a "generative game engine" could also go offroad, extrapolating what would happen if you go to unseen places. I'd even say extrapolation capabilities of such a model could be better than a traditional game engine, as it can make things up as it goes, while if you accidentally cross a wall in a typical game engine the screen goes blank.

Re: Diffusion models are real-time game engines

#137

Post by godelski »

snickmy wrote: post #129
godelski wrote: post #125

Doom system requirements: - 4 MB RAM - 12 MB disk space Stable diffusion v1 > 860M UNet and CLIP ViT-L/14 (540M) Checkpoint size: 4.27 Gb 7.7 GB (full EMA) Running on a TPU-v5e Peak compute per chip (bf16) 197 TFLOPs Peak compute per chip (Int8) 393 TFLOPs HBM2 capacity and bandwidth 16 GB, 819 GBps Interchip Interconnect BW 1600 Gbps This is quite impressive, especially considering the speed. But there's still a ton…

Those are valid points, but irrelevant for the context of this research. Yes, the computational cost is ridicolous compared to the original game, and yes, it lacks basic things like pre-computing, storing, etc. That said, you could assume that all that can be either done at the margin of this discovery OR over time will naturally improve OR will become less important as a blocker. The fact that you can model a sequen…

I'm not sure what you're saying is irrelevant.

1) the model has enough memory to store not only all game assets and engine but even hundreds of "plays".

2) me mentioning that there's still a lot of room to make these things better (seems you think so too so maybe not this one?)

3) an interesting point I was wondering to compare current state of things (I mean I'll give you this but it's just a random thought and I'm not reviewing this paper in an academic setting. This is HN, not NeurIPS. I'm just curious ¯ \ _ ( ツ ) _ / ¯)

4) the point that you can rip a game

I'm really not sure what you're contesting to because I said several things.

  > it lacks basic things like pre-computing, storing, etc.
It does? Last I checked neural nets store information. I guess I need to return my PhD because last I checked there's a UNet in SD 1.4 and that contains a decoder.

Re: Diffusion models are real-time game engines

#138

Post by jamilton »

jamilton wrote: post #98

I wonder if the MineRL ( https://www.ijcai.org/proceedings/2019/0339.pdf and minerl.io) dataset would be sufficient to reproduce this work with Minecraft. Any other similar existing datasets? A really goofy way I can think of to get a bunch of data would be to get videos from youtube and try to detect keyboard sounds to determine what keys they're pressing.

Although ideally a follow up work would be something where there won’t be any potential legal trouble with releasing the complete model so people can play it.

A similar approach but with a game where the exact input is obvious and unambiguous from the graphics alone so that you can use unannotated data might work. You’d just have to create a model to create the action annotations. I’m not sure what the point would be, but it sounds like it’d be interesting.

Re: Diffusion models are real-time game engines

#139

Post by meheleventyone »

panki27 wrote: post #120

> Human raters are only slightly better than random chance at distinguishing short clips of the game from clips of the simulation. I can hardly believe this claim, anyone who has played some amount of DOOM before should notice the viewport and textures not "feeling right", or the usually static objects moving slightly.

It's telling IMO that they only want people opinions based on our notoriously faulty memories rather than sitting comparable situations next to one another in the game and simulation then analyzing them. Several things jump out watching the example video.

Re: Diffusion models are real-time game engines

#140

Post by pickledoyster »

snickmy wrote: post #129
godelski wrote: post #125

Doom system requirements: - 4 MB RAM - 12 MB disk space Stable diffusion v1 > 860M UNet and CLIP ViT-L/14 (540M) Checkpoint size: 4.27 Gb 7.7 GB (full EMA) Running on a TPU-v5e Peak compute per chip (bf16) 197 TFLOPs Peak compute per chip (Int8) 393 TFLOPs HBM2 capacity and bandwidth 16 GB, 819 GBps Interchip Interconnect BW 1600 Gbps This is quite impressive, especially considering the speed. But there's still a ton…

Those are valid points, but irrelevant for the context of this research. Yes, the computational cost is ridicolous compared to the original game, and yes, it lacks basic things like pre-computing, storing, etc. That said, you could assume that all that can be either done at the margin of this discovery OR over time will naturally improve OR will become less important as a blocker. The fact that you can model a sequen…

>you could assume that all that can be either done at the margin of this discovery OR over time will naturally improve OR will become less important as a blocker.

OR one can hope it will be thrown to the heap of nonviable tech with the rest of spam waste

Post reply on HN