Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Diffusion models are real-time game engines

gamengen.github.io

Post reply on HN
Print view View original

421–430 of 430 posts

Re: Diffusion models are real-time game engines

#421

Post by slashdave »

Sharlin wrote: post #410
slashdave wrote: post #56

Earlier quoted context omitted.

It is just video. There are no external interactions. Heck, it is far simpler than video, because the point of view and frame is fixed.

Uff, I guess you’re right. Mea culpa. I misread their diagram to represent inference when it was about training instead. The latter is conditioned on actions, but… how do they generate the actual output frames then? What’s the input? Is it just image-to-image based on the previous frame? The paper doesn’t seem to explain the inference part at all well :(

It should be possible to generate an initial image from Gaussian noise, including the latent information on player position

Re: Diffusion models are real-time game engines

#422

Post by dTal »

godelski wrote: post #420
dTal wrote: post #165

Earlier quoted context omitted.

>What's also interesting about this work is it's basically saying you can rip a game if you're willing to "play" (automate) it enough times and spend a lot more on storage and compute That's the least of it. It means you can generate a game from real footage. Want a perfect flight sim? Put a GoPro in the cockpit of every airliner for a year.

> Want a perfect flight sim? Put a GoPro in the cockpit of every airliner for a year. You're jumping ahead there and I'm not convinced you could do this ever (unless you're model is already a great physics engine). The paper itself has feeds the controls into the network. But a flight sim will be harder better you'd need to also feed in air conditions. I just don't see how you could do this from video alone, let alon…

There's an enormous amount of information if your GoPro placement includes all the flight instruments. Humans can and do predict aircraft state t+1 by parsing a visual field that includes the instruments; that is what the instruments are for.

Re: Diffusion models are real-time game engines

#423

Post by cthalupa »

bee_rider wrote: post #235
rvnx wrote: post #225

Earlier quoted context omitted.

This was the basis for the reasoning: The map 1 has 2'518 walkable map units. There are 65536 angles. 2'518*65'536=165'019'648 If you capture 165M frames, you already cover all the possibilities in terms of camera / player view, but probably the diffusion models don't even need to have all the frames (the same way that LLMs don't).

Do you have to be exactly on a tile in Doom? I thought the guy walked smoothly around the map.

> I thought the guy walked smoothly around the map.

Correct. You are certainly not moving between the tiles as discrete units in doom.

Re: Diffusion models are real-time game engines

#424

Post by HellDunkel »

zamadatix wrote: post #417
HellDunkel wrote: post #118

Although impressive i must disagree. Diffusion models are not game engines. A game engine is a component to propell your game (along the time axis?). In that sense it is similar to the engine of the car, hence the name. It does not need a single working car nor a road to drive on do its job. The above is a dynamic, interactive replication of what happens when you put a car on a given road, requiring a million test dr…

This seems more a critique on the particular model (as in the resulting diffusion model generated), not on diffusion models in general. It's also a bit misstated - this doesn't require a working car on the road to do its job (present tense), it required one to train it to do its job (past tense) and it's not particularly clear why a game engine using concepts gained from how another worked should cease to be a game e…

my point is something else: a game engine is something which can be separated from a game and put to use somewhere else. this is basically the definition of „engine“. the above is not an engine but a game without any engine at all therefor should not be called „engine“.

Re: Diffusion models are real-time game engines

#425

Post by BizarroLand »

whiteboardr wrote: post #267
beepbooptheory wrote: post #245

Earlier quoted context omitted.

What would the model provide if not what we see on the screen?

The environment and everything in it. “Everything” would mean all objects and the elements they’re made of, their rules on how they interact and decay. A modularized ecosystem i guess, comprised of “sub-systems” of sorts. The other model, that provides all interaction (cause for effect) could either be run artificially or be used interactively by a human - opening up the possibility for being a tree : ) This all woul…

For instance, for a generated real world RPG, one process could create the planet, one could create the city where the player starts, one could create the NPCs, one could then model the relationships of the npcs with each other. Each one building off of the other so that the whole thing feels nuanced and more real.

Repeat for quest lines, new cities, etc, with the npcs having real time dialogue and interactions that happen entirely off screen, no guarantee of there being a massive quest objective, and some sort of recorder of events that keeps a running tally of everything that goes on so that as the PCs interact with it they are never repeating the same dreary thing.

If this were a MMORPG it would require so much processing and architecting, but it would have the potential to be the greatest game in human history.

Re: Diffusion models are real-time game engines

#426

Post by nuancebydefault »

Sohcahtoa82 wrote: post #329

It's always fun reading the dead comments on a post like this. People love to point how how pointless this is. Some of ya'll need to learn how to make things for the fun of making things . Is this useful? No, not really. Is it interesting? Absolutely. Not everything has to be made for profit. Not everything has to be made to make the world a better place. Sometimes, people create things just for the learning experien…

Today this exercise feels pointless. However. I remember the days when there were articles written about the possibility for "internet radio". In stead of good old broadcasting waves through the ether and simply thousands of radios tuning in, some server would send a massive amount of packets over a massive kilometers of copper to thousands of endpoints. Ad absurdum the endpoints would even send ack packages upstream to the poor server to keep connections alive. It seemed like a huge waste of computing power, wire and energy.

And here we are, binging netflix movies over such copper wires.

I'm not saying games will be replaced by diffusion models dreaming up next images based on user input, but a variation of that might end up in a form of interactive art creation or a new form of entertainment.

Re: Diffusion models are real-time game engines

#427

Post by harha_ »

aithrowaway1987 wrote: post #293
harha_ wrote: post #282

This is so sick I don't know what to say. I never expected this, aren't the implications of this huge?

I am struggling to understand a single implication of this! How does this generalize to anything other than other than playing retro games in the most expensive way possible? The very intention of this project is overfitting to data in a non-generalizable way! Maybe it's just pure engineering, that good ANNs are getting cheap and fast. But this project still seems to have the fundamental weaknesses of all AI projects…

I agree with you, when I made this comment I was simply excited but that didn't last too long. I find this technology both exciting and dystopian, the latter because the dystopic use of it is already happening all over the internet. For now, it's been used only for entertainment AFAIK, which is the kind of use I don't like either, because I prefer human created entertainment over this crap.

Re: Diffusion models are real-time game engines

#428

Post by hypertele-Xii »

Sharlin wrote: post #214
hypertele-Xii wrote: post #198

Earlier quoted context omitted.

Then why do monsters become blurry smudgy messes when shot? That looks like a video compression artifact of a neural network attempting to replicate low-structure image (source material contains guts exploding, very un-structured visual).

Uh, maybe because monster death animations make up a small part of the training material (ie. gameplay) so the model has not learned to reproduce them very well? There cannot be "video compression artifacts" because it hasn’t even seen any compressed video during training, as far as I can see. Seriously, how is this even a discussion? The article is clear that the novel thing is that this is real-time frame generatio…

I simply cannot take seriously anyone who exclaims that monster death animations are a minor part of Doom. It's literally a game about slaying demons. Gameplay consists almost entirely of explosions and gore, killing monsters IS THE GAME, if you can't even get that correct then what nonsense are we even looking at.

Re: Diffusion models are real-time game engines

#430

Post by w_for_wumbo »

troupo wrote: post #88
w_for_wumbo wrote: post #13

Earlier quoted context omitted.

That feels like the endgame of video game generation. You select an art style, a video and the type of game you'd like to play. The game is then generated in real-time responding to each action with respect to the existing rule engine. I imagine a game like that could get so convincing in its details and immersiveness that one could forget they're playing a game.

There are thousands of games that mimic each other, and only a handful of them are any good. What makes you think a mechanical "predict next frame based on existing games" will be any good?

Oh, because we can link this in with biometric responses - heartrate, temperature, eye tracking etc.

We could build a 'game' which would learn and adapt to precisely the chemistry that makes someone tick and then provide them a map to find the state in which their brain releases their desired state.

Then if the game has a directive - it should be pointed to work as a training tool to allow the user to determine how to release these chemicals themselves at will. Resulting in a player-base which no longer requires anything external for accessing their own desired states.

Post reply on HN