What if instead of a video game, this was trained on video and control inputs from people operating equipment like warehouse robots? Then an automated system could visualize the result of a proposed action or series of actions when operating the equipment itself. You would need a different model/algorithm to propose control inputs, but this would offer a way for the system to validate and refine plans as part of a problem solving feedback loop.
Diffusion models are real-time game engines
261–270 of 430 posts
Re: Diffusion models are real-time game engines
#262Earlier quoted context omitted.
> Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected To temper this a bit, you may want to pay close attention to the demo videos. The player rarely backtracks, and for good reason - the few times the character does turn around and look back at something a second time, it has changed significantly (the most noticeable I think is the room with the grey wall and triangle sign…
I don't see this as something that would be hard to overcome. Sora for instance has already shown the ability for a diffusion model to maintain object permanence. Flux recently too has shown the ability to render the same person in many different poses or images.
Where does a sora video turn around backwards? I don’t even maintain such consistency in my dreams.
Re: Diffusion models are real-time game engines
#263Earlier quoted context omitted.
I saw a longer video of this that Ethan Mollick posted and in that one, the sequences are longer and they do appear to demonstrate a fair amount of consistency. The clips don't backtrack in the summary video on the paper's home page because they're showing a number of district environments but you only get a few seconds of each. If I studied the longer one more closely, I'm sure inconsistencies would be seen but it s…
Yes it definitely is very good for simulating gameplay footage, don't get me wrong. Its input for predicting the next frame is not just the previous frame, it has access to a whole sequence of prior frames. But to say the model is simulating actual gameplay (i.e. that a person could actually play Doom in this) is far fetched. It's definitely great that the model was able to remember that the gray wall was still there…
> it's untenable for actual gameplay that the wall completely changed location and orientation.
It would in an SCP-themed game. Or dreamscape/Inception themed one.
Hell, "you're trapped in Doom-like dreamscape, escape before you lose your mind" is a very interesting pitch for a game. Basically take this Doom thing and make walking though a specific, unique-looking doorway from the original game to be the victory condition - the player's job would be to coerce the model to generate it, while also not dying in the Doom fever dream game itself. I'd play the hell out of this.
(Implementation-wise, just loop in a simple recognition model to continously evaluate victory condiiton from last few frames, and some OCR to detect when player's hit points indicator on the HUD drops to zero.)
(I'll happily pay $100 this year to the first project that gets this to work. I bet I'm not the only one. Doesn't have to be Doom specifically, just has to be interesting.)
Re: Diffusion models are real-time game engines
#264So, this is surprising. Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected, which would be roughly ‘none’. Google here uses SD 1.4, as the core of the diffusion model, which is a nice reminder that open models are useful to even giant cloud monopolies. The two main things of note I took away from the summary were: 1) they got infinite training data using agents playing doom…
Nicely summarised. Another important thing that clearly standsout (not to undermine the efforts and work gone into this) is the fact that more and more we are now seeing larger and more complex building blocks emerging (first it was embedding models then encoder decoder layers and now whole models are being duck-taped for even powerful pipelines). AI/DL ecosystem is growing on a nice trajectory.
Though I wonder if 10 years down the line folks wouldn't even care about underlying model details (no more than a current day web-developer needs to know about network packets).
PS: Not great examples, but I hope you get the idea ;)
Re: Diffusion models are real-time game engines
#265Earlier quoted context omitted.
> Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected To temper this a bit, you may want to pay close attention to the demo videos. The player rarely backtracks, and for good reason - the few times the character does turn around and look back at something a second time, it has changed significantly (the most noticeable I think is the room with the grey wall and triangle sign…
I don't see this as something that would be hard to overcome. Sora for instance has already shown the ability for a diffusion model to maintain object permanence. Flux recently too has shown the ability to render the same person in many different poses or images.
Where does a sora video turn around backwards? I can’t maintain such consistency in my own dreams.
Re: Diffusion models are real-time game engines
#266Earlier quoted context omitted.
1 billion frames in memory... With such dataset, you have seen practically all realistic possibilities in the short-term. If it would be able to invent action and maps and let the user play "infinite doom", then it would be very different (and impressive!).
Like many people in case of LLMs, you're just demonstrating unawareness of - or disbelief in - the fact that the model doesn't record training data vetbatim, but smears it out in high-dimensional space, from which it then samples. The model then doesn't recall past inputs (which are effectively under extreme lossy compression), but samples from that high-dimensional space to produce output. The high-dimensional repre…
Whether or not a judge understands this will probably form the basis of any precedent set about the legality of image models and copyright.
Re: Diffusion models are real-time game engines
#267Earlier quoted context omitted.
But does it need to be frame-based? What if you combine this with an engine in parallel that provides all geometry including characters and objects with their respective behavior, recording changes made through interactions the other model generates, talking back to it? A dialogue between two parties with different functionality so to speak. (Non technical person here - just fantasizing)
What would the model provide if not what we see on the screen?
The environment and everything in it.
“Everything” would mean all objects and the elements they’re made of, their rules on how they interact and decay.
A modularized ecosystem i guess, comprised of “sub-systems” of sorts.
The other model, that provides all interaction (cause for effect) could either be run artificially or be used interactively by a human - opening up the possibility for being a tree : )
This all would need an interfacing agent that in principle would be an engine simulating the second law of thermodynamics and at the same time recording every state that has changed and diverged off the driving actor’s vector in time.
Basically the “effects” model keeping track of everyones history.
In the end a system with an “everything” model (that can grow overtime), a “cause” model messing with it, brought together and documented by the “effect” model.
(Again … non technical person, just fantasizing) : )
Re: Diffusion models are real-time game engines
#268[dead]
Re: Diffusion models are real-time game engines
#269Earlier quoted context omitted.
No, it implements a 3D FPS that's interactive, and renders each frame based on your input and a lot of memorized gameplay.
But is it playing the actual game or just making a interactive video of it?
Making an interactive video of it. It is not playing the game, a human does that.
With that said, I wholly disagree that this is not an engine. This is absolutely a game engine and while this particular demo uses the engine to recreate DOOM, an existing game, you could certainly use this engine to produce new games in addition to extrapolating existing games in novel ways.
Re: Diffusion models are real-time game engines
#270Earlier quoted context omitted.
> Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected To temper this a bit, you may want to pay close attention to the demo videos. The player rarely backtracks, and for good reason - the few times the character does turn around and look back at something a second time, it has changed significantly (the most noticeable I think is the room with the grey wall and triangle sign…
Great observation. And not entirely unlike normal human visual perception which is notoriously vulnerable to missing highly salient information; I'm reminded of the "gorillas in our midst" work by Dan Simons and Christopher Chabris [0]. [0]: https://en.wikipedia.org/wiki/Inattentional_blindness#Invisi...
Are you saying if I turn around, I’ll be surprised at what I find ? I don’t feel like this is accurate at all.