This is insanely good but look at the legs around 16 seconds in, they kinda morph through each other. Generally the legs are slightly unnerving.
Still, god damn.
Video generation models as world simulators
11–20 of 171 posts
Re: Video generation models as world simulators
#12The improvement to temporal consistency given that the length of these generated videos is 3 to 4 times longer than anything else on the market (runway, pika, etc) is truly remarkable.
Re: Video generation models as world simulators
#13I like that this one shows some "fails", and not just the top of the top results:
For example, the surfer is surfing in the air at the end:
https://cdn.openai.com/tmp/s/prompting_7.mp4
Or this "breaking" glass that does not break, but spills liquid in some weird way:
https://cdn.openai.com/tmp/s/discussion_0.mp4
Or the way this person walks:
https://cdn.openai.com/tmp/s/a-woman-wearing-a-green-dress-a...
Or wherever this map is coming from:
https://cdn.openai.com/tmp/s/a-woman-wearing-purple-overalls...
Re: Video generation models as world simulators
#14Wow, this is really just scale-up DiT. We are going to see tons of similar models very soon.
Re: Video generation models as world simulators
#15I find it wild that this model does not have explicit 3D prior, yet learns to generate videos with such 3D consistency, you can directly train a 3D representation (NeRF-like) from those videos: https://twitter.com/BenMildenhall/status/1758224827788468722
I wonder how much it'd improve if trained on stereo image data.
Re: Video generation models as world simulators
#16I find it wild that this model does not have explicit 3D prior, yet learns to generate videos with such 3D consistency, you can directly train a 3D representation (NeRF-like) from those videos: https://twitter.com/BenMildenhall/status/1758224827788468722
I was similarly astonished at this adaptation of stable diffusion to make HDR spherical environment maps from existing images- https://diffusionlight.github.io/ The crazy thing is that they do it by prompting the model to in paint a chrome sphere into the middle of the image to reflect what is behind the camera! The model can interpret the context and dream up what is plausibly in the whole environment.
Yeah we were surprised by that, video models are great 3d prices and image models are great video model priors
Re: Video generation models as world simulators
#17I think people might be missing what this enables. It can make plausible continuations of video, with realistic physics. What happens if this gets fast enough to work _in real time_.
Connect this to a robot that has a real time camera feed. Have it constantly generate potential future continuations of the feed that it's getting -- maybe more than one. You have an autonomous robot building a real time model of the world around it and predicting the future. Give it some error correction based on well each prediction models the actual outcome and I think you're _really_ close to AGI.
You can probably already imagine different ways to wire the output to text generation and controlling its own motions, etc, and predicting outcomes based on actions it, itself could plausibly take, and choosing the best one.
It doesn't actually have to generate realistic imagery or imagery that doesn't have any mistakes or imagery that's high definition to be used in that way. How realistic is our own imagination of the world?
Edit: I'm going to add a specific case. Imagine a house cleaning robot. It starts with an image of your living room. Then it creates a image of your living room after it's been cleaned. Then it interpolates a video _imagining itself cleaning the room_, then acts as much as it can to mimic what's in the video, then generates a new continuation, then acts, and so on. Imagine doing that several times a second, if necessary.
Re: Video generation models as world simulators
#18That's an interesting idea. Analogous to how LLMs are simply "text predictors" but end up having to learn a model of language and the world to correctly predict cohesive text, it makes sense that "video predictors" also have to learn a model of the world that makes sense. I wonder how many orders of magnitude further they have to evolve to be similarly useful.
Re: Video generation models as world simulators
#19Damn, even minecraft videos being simulated, this is crazy to see from OpenAI. Edit, changed the links to the direct ones! https://cdn.openai.com/tmp/s/simulation_6.mp4 https://cdn.openai.com/tmp/s/simulation_7.mp4
But it starts to make sense, when you think about the fact that Minecraft is owned by Microsoft. example video links from TFA: https://cdn.openai.com/tmp/s/simulation_6.mp4 https://cdn.openai.com/tmp/s/simulation_7.mp4
I don't think that has anything to do with it. There's just millions of hours of minecraft video on youtube and twitch.
Re: Video generation models as world simulators
#20I think people might be missing what this enables. It can make plausible continuations of video, with realistic physics. What happens if this gets fast enough to work _in real time_. Connect this to a robot that has a real time camera feed. Have it constantly generate potential future continuations of the feed that it's getting -- maybe more than one. You have an autonomous robot building a real time model of the wor…
Sounds like simulation theory is closer and closer to being proven.