Video generation models as world simulators
1–10 of 171 posts
Re: Video generation models as world simulators
#2I find it wild that this model does not have explicit 3D prior, yet learns to generate videos with such 3D consistency, you can directly train a 3D representation (NeRF-like) from those videos: https://twitter.com/BenMildenhall/status/1758224827788468722
Re: Video generation models as world simulators
#3I find it wild that this model does not have explicit 3D prior, yet learns to generate videos with such 3D consistency, you can directly train a 3D representation (NeRF-like) from those videos: https://twitter.com/BenMildenhall/status/1758224827788468722
This is also true for 2D diffusion models[1]. I suppose they need to understand how 3d works for stuff like lighting/shadows/object occlusion, etc.
Re: Video generation models as world simulators
#4I find it wild that this model does not have explicit 3D prior, yet learns to generate videos with such 3D consistency, you can directly train a 3D representation (NeRF-like) from those videos: https://twitter.com/BenMildenhall/status/1758224827788468722
That leaves me wondering if it'd be possible to get some variant of the model to directly output 3D meshes and camera animation instead of an image.
Re: Video generation models as world simulators
#5I know the main post has been getting a lot of reaction, but this page absolutely blew me away. The results are striking.
The robot examples are very underwhelming, but the people and background people are all very well done, and at a level much better than most static image diffusion models produce. Generating the same people as the interact with objects is also not something I expected a model like this to do well so soon.
Re: Video generation models as world simulators
#6Damn, even minecraft videos being simulated, this is crazy to see from OpenAI.
Edit, changed the links to the direct ones!
Re: Video generation models as world simulators
#7AlphaGo and AlphaZero were able to achieve superhuman performance due to the availability of perfect simulators for the game of Go. There is no such simulator for the real world we live in (although pure LLMs sort of learn a rough, abstract representation of the world as perceived by humans.) Sora is an attempt to build such a simulator using deep learning.
“Our results suggest that scaling video generation models is a promising path towards building general purpose simulators of the physical world.”
Why superhuman? Larger context length than our working memory is an obvious one, but there will likely be other advantages such as using alternative sensory modalities and more granular simulation of details unfamiliar to most humans.
Re: Video generation models as world simulators
#8If there's one thing I've always wanted, it's shitty video knockoffs of real life. Can't wait to stream some AI hallucinations.
Re: Video generation models as world simulators
#9Damn, even minecraft videos being simulated, this is crazy to see from OpenAI. Edit, changed the links to the direct ones! https://cdn.openai.com/tmp/s/simulation_6.mp4 https://cdn.openai.com/tmp/s/simulation_7.mp4
But it starts to make sense, when you think about the fact that Minecraft is owned by Microsoft.
example video links from TFA:
Re: Video generation models as world simulators
#10I find it wild that this model does not have explicit 3D prior, yet learns to generate videos with such 3D consistency, you can directly train a 3D representation (NeRF-like) from those videos: https://twitter.com/BenMildenhall/status/1758224827788468722
I was similarly astonished at this adaptation of stable diffusion to make HDR spherical environment maps from existing images- https://diffusionlight.github.io/
The crazy thing is that they do it by prompting the model to in paint a chrome sphere into the middle of the image to reflect what is behind the camera! The model can interpret the context and dream up what is plausibly in the whole environment.