Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Diffusion models are real-time game engines

gamengen.github.io

Post reply on HN
Print view View original

281–290 of 430 posts

Re: Diffusion models are real-time game engines

#281

Post by matheusd »

bamboozled wrote: post #270
dewarrn1 wrote: post #221

Earlier quoted context omitted.

Great observation. And not entirely unlike normal human visual perception which is notoriously vulnerable to missing highly salient information; I'm reminded of the "gorillas in our midst" work by Dan Simons and Christopher Chabris [0]. [0]: https://en.wikipedia.org/wiki/Inattentional_blindness#Invisi...

Are you saying if I turn around, I’ll be surprised at what I find ? I don’t feel like this is accurate at all.

If a generic human glances at an unfamiliar screen/wall/room, can they accurately, pixel-perfectly reconstruct every single element of it? Can they do it for every single screen they have seen in their entire lives?

Re: Diffusion models are real-time game engines

#283

Post by robotresearcher »

whiteboardr wrote: post #217
wavemode wrote: post #190

Earlier quoted context omitted.

> Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected To temper this a bit, you may want to pay close attention to the demo videos. The player rarely backtracks, and for good reason - the few times the character does turn around and look back at something a second time, it has changed significantly (the most noticeable I think is the room with the grey wall and triangle sign…

But does it need to be frame-based? What if you combine this with an engine in parallel that provides all geometry including characters and objects with their respective behavior, recording changes made through interactions the other model generates, talking back to it? A dialogue between two parties with different functionality so to speak. (Non technical person here - just fantasizing)

In that scheme what is the NN providing that a classical renderer would not? DOOM ran great on an Intel 486, which is not a lot of computer.

Re: Diffusion models are real-time game engines

#284

Post by znx_0 »

TeMPOraL wrote: post #257
rvnx wrote: post #208

Earlier quoted context omitted.

1 billion frames in memory... With such dataset, you have seen practically all realistic possibilities in the short-term. If it would be able to invent action and maps and let the user play "infinite doom", then it would be very different (and impressive!).

Like many people in case of LLMs, you're just demonstrating unawareness of - or disbelief in - the fact that the model doesn't record training data vetbatim, but smears it out in high-dimensional space, from which it then samples. The model then doesn't recall past inputs (which are effectively under extreme lossy compression), but samples from that high-dimensional space to produce output. The high-dimensional repre…

I like "conditioned brute force" better term.

Re: Diffusion models are real-time game engines

#285

Post by ValentinA23 »

wavemode wrote: post #279
TeMPOraL wrote: post #263

Earlier quoted context omitted.

> it's untenable for actual gameplay that the wall completely changed location and orientation. It would in an SCP-themed game. Or dreamscape/Inception themed one. Hell, "you're trapped in Doom-like dreamscape, escape before you lose your mind" is a very interesting pitch for a game. Basically take this Doom thing and make walking though a specific, unique-looking doorway from the original game to be the victory cond…

To be honest, I agree! That would be an interesting gameplay concept for sure. Mainly just wanted to temper expectations I'm seeing throughout this thread that the model is actually simulating Doom. I don't know what will be required to get from here to there, but we're definitely not there yet.

What you're pointing at mirrors the same kind of limitation in using LLMs for role-play/interactive fictions.

Re: Diffusion models are real-time game engines

#286

Post by whiteboardr »

robotresearcher wrote: post #283
whiteboardr wrote: post #217

Earlier quoted context omitted.

But does it need to be frame-based? What if you combine this with an engine in parallel that provides all geometry including characters and objects with their respective behavior, recording changes made through interactions the other model generates, talking back to it? A dialogue between two parties with different functionality so to speak. (Non technical person here - just fantasizing)

In that scheme what is the NN providing that a classical renderer would not? DOOM ran great on an Intel 486, which is not a lot of computer.

An experience that isn’t asset- but rule-based.

Re: Diffusion models are real-time game engines

#287

Post by bamboozled »

matheusd wrote: post #281
bamboozled wrote: post #270

Earlier quoted context omitted.

Are you saying if I turn around, I’ll be surprised at what I find ? I don’t feel like this is accurate at all.

If a generic human glances at an unfamiliar screen/wall/room, can they accurately, pixel-perfectly reconstruct every single element of it? Can they do it for every single screen they have seen in their entire lives?

I never said pixel perfect, but I would be surprised if whole objects , like flaming lanterns suddenly appeared.

What this demo demonstrates to me is how incredible willing we are to accept what seems familiar to us as accurate.

I bet if you look closely and objectively you will see even more anomalies. But at first watch, I didn’t see most errors because I think accepting something is more efficient for the brain.

Re: Diffusion models are real-time game engines

#288

Post by PunchTornado »

nsbk wrote: post #181
wkcheng wrote: post #9

It's insane that that this works, and that it works fast enough to render at 20 fps. It seems like they almost made a cross between a diffusion model and an RNN, since they had to encode the previous frames and actions and feed it into the model at each step. Abstractly, it's like the model is dreaming of a game that it played a lot of, and real time inputs just change the state of the dream. It makes me wonder if hu…

We are. At least that's what Lisa Feldman Barrett [1] thinks. It is worth listening to this Lex Fridman podcast: Counterintuitive Ideas About How the Brain Works [2], where she explains among other ideas how constant prediction is the most efficient way of running a brain as opposed to reaction. I never get tired of listening to her, she's such a great science communicator. [1] https://en.wikipedia.org/wiki/Lisa_Feld…

Interesting talk about the brain, but the stuff she says about free will is not a very good argument. Basically it is sort of the argument that the ancient greeks made which brings the discussion into a point where you can take both directions.

Re: Diffusion models are real-time game engines

#289

Post by KajMagnus »

wavemode wrote: post #279
TeMPOraL wrote: post #263

Earlier quoted context omitted.

> it's untenable for actual gameplay that the wall completely changed location and orientation. It would in an SCP-themed game. Or dreamscape/Inception themed one. Hell, "you're trapped in Doom-like dreamscape, escape before you lose your mind" is a very interesting pitch for a game. Basically take this Doom thing and make walking though a specific, unique-looking doorway from the original game to be the victory cond…

To be honest, I agree! That would be an interesting gameplay concept for sure. Mainly just wanted to temper expectations I'm seeing throughout this thread that the model is actually simulating Doom. I don't know what will be required to get from here to there, but we're definitely not there yet.

Or if training the model on many FPS games? Surviving in one nightmare that morphs into another, into another, into another ...

Re: Diffusion models are real-time game engines

#290

Post by dewarrn1 »

robotresearcher wrote: post #280
dewarrn1 wrote: post #221

Earlier quoted context omitted.

Great observation. And not entirely unlike normal human visual perception which is notoriously vulnerable to missing highly salient information; I'm reminded of the "gorillas in our midst" work by Dan Simons and Christopher Chabris [0]. [0]: https://en.wikipedia.org/wiki/Inattentional_blindness#Invisi...

Not noticing to a gorilla that ‘shouldn’t’ be there is not the same thing as object permanence. Even quite young babies are surprised by objects that go missing.

That's absolutely true. It's also well-established by Simons et al. and others that healthy normal adults maintain only a very sparse visual representation of their surroundings, anchored but not perfectly predicted by attention, and this drives the unattended gorilla phenomenon (along with many others). I don't work in this domain, but I would suggest that object permanence probably starts with attending and perceiving an object, whereas the inattentional or change blindness phenomena mostly (but not exclusively) occur when an object is not attended (or only briefly attended) or attention is divided by some competing task.
Post reply on HN