Should I short all the 3d tool s/movies/vfx companies?
Video generation models as world simulators
21–30 of 171 posts
Re: Video generation models as world simulators
#22I think people might be missing what this enables. It can make plausible continuations of video, with realistic physics. What happens if this gets fast enough to work _in real time_. Connect this to a robot that has a real time camera feed. Have it constantly generate potential future continuations of the feed that it's getting -- maybe more than one. You have an autonomous robot building a real time model of the wor…
Sounds like simulation theory is closer and closer to being proven.
Our ability to build somewhat convincing simulations of thing has never been a proof of living in a simulation…
Re: Video generation models as world simulators
#23Earlier quoted context omitted.
Sounds like simulation theory is closer and closer to being proven.
Our ability to build somewhat convincing simulations of thing has never been a proof of living in a simulation…
i mean everyone's mind builds a convincing internal simulation of reality and it's so good that most people think they're directly experiencing reality.
Re: Video generation models as world simulators
#24I think it's Ylecun who stated a few times that video was the better way to train large models as it's more information dense.
The results really are impressive. Being able to generate such high quality videos, to extend videos in the past and in the future shows how much the model "understands" the real world, objects interaction, 3D composition, etc...
Although image generation already requires the model to know a lot about the world, i think there's really a huge gap with video generation where the model needs to "know" 3D, objects movements and interactions.
Re: Video generation models as world simulators
#25I think people might be missing what this enables. It can make plausible continuations of video, with realistic physics. What happens if this gets fast enough to work _in real time_. Connect this to a robot that has a real time camera feed. Have it constantly generate potential future continuations of the feed that it's getting -- maybe more than one. You have an autonomous robot building a real time model of the wor…
Sounds like simulation theory is closer and closer to being proven.
Except there is always an original at the root. There’s no way to prove that’s not us.
Re: Video generation models as world simulators
#26The Minecraft demo makes me think that soon will be playing games directly from the output of one of these models, unlimited content.
Re: Video generation models as world simulators
#27I think people might be missing what this enables. It can make plausible continuations of video, with realistic physics. What happens if this gets fast enough to work _in real time_. Connect this to a robot that has a real time camera feed. Have it constantly generate potential future continuations of the feed that it's getting -- maybe more than one. You have an autonomous robot building a real time model of the wor…
The flip side of video or image gen is always video or image identification. If video gets really good then an AI can have quite an accurate visual view into the world in real time
Re: Video generation models as world simulators
#28Should I short all the 3d tool s/movies/vfx companies?
I think not in the short term- I'm guessing the next step will be to use traditional tools to make a "draft" of a desired video, then "finish it off" with this kind of deep learning tech.
So this tech will increase interest in existing 3d tools in the short term
Re: Video generation models as world simulators
#29I like that this one shows some "fails", and not just the top of the top results: For example, the surfer is surfing in the air at the end: https://cdn.openai.com/tmp/s/prompting_7.mp4 Or this "breaking" glass that does not break, but spills liquid in some weird way: https://cdn.openai.com/tmp/s/discussion_0.mp4 Or the way this person walks: https://cdn.openai.com/tmp/s/a-woman-wearing-a-green-dress-a... Or wherever…
The hyper realistic and plausible movement of the glass breaking makes this bizarrely fascinating. And it doesn’t give me the feeling of disgust the motion in the more primitive AI models did
Re: Video generation models as world simulators
#30I think people might be missing what this enables. It can make plausible continuations of video, with realistic physics. What happens if this gets fast enough to work _in real time_. Connect this to a robot that has a real time camera feed. Have it constantly generate potential future continuations of the feed that it's getting -- maybe more than one. You have an autonomous robot building a real time model of the wor…
Thanks for adding the specific case. I think with testing these sort of limited domain applications make sense.
It'll be much harder for more open ended world problems where the physics encountered may be rare enough in the dataset that the simulation breaks unexpectedly. For example a glass smashing into the floor. The model doesn't simulate that causally afaik