very cool - what was the hardest part of building this?
AI video you can watch and interact with, in real-time
41–50 of 91 posts
Re: AI video you can watch and interact with, in real-time
#42[deleted]
Re: AI video you can watch and interact with, in real-time
#43Earlier quoted context omitted.
Only at first glance. It can easily render things that would be very hard to implement in an FPS engine. What AI can dream up in milliseconds could take hundreds of human hours to encode using traditional tech (meshes, shaders, ray tracing, animation, logic scripts, etc.), and it still wouldn't look as natural and smooth as AI renderings — I refer to the latest developments in video synthesis like Google's Veo 3. Ima…
Why do you think this is so hard, even for technical people here, to make the inductive leap on this one? Is it that close to magic? The AI is rendering pillars and also determining collision detection on it. As in, no one went in there and selected a bunch of pillars and marked it as a barrier. That means in the long run, I'll be able to take some video or pictures of the real world and have it be game level.
Because that's been a thing for years already - and works way better then this research does.
Unreal engine 5 has been demoing these features for a while now, I heard about it early 2020 iirc, but the techniques like gaussian splattering predate it.
I have no experience in either of these, but I believe MegaScans and RealityCapture are two examples doing this. And the last nanite demo touched on it, too.
Re: AI video you can watch and interact with, in real-time
#44This is pretty much the same thing as those models that baked dust2 into a diffusion model then used the last few frames as context to continue generating - same failure modes and everything.
Re: AI video you can watch and interact with, in real-time
#45[dead]
Re: AI video you can watch and interact with, in real-time
#46That felt so wrong AND someone is cheating here. This felt really suspicious... I got to the graffiti world and there were some stairs right next to me. So I started going up them. It felt like I was walking forward and the stairs were pushing under me until I just got stuck. So I turned to go back down and half way around everything morphed and I ended up back down at the ground level where I originally was. I was t…
It’s essentially this paper but applied to a bunch of video recordings of a bunch of different real world locations instead of counter strike maps. Each channel is just changing the location.
Re: AI video you can watch and interact with, in real-time
#47Would be more interesting with people in it.
Re: AI video you can watch and interact with, in real-time
#48Note that it isn't being created from whole cloth, it is trained on videos of the places and then it is generating the frames: "To improve autoregressive stability for this research preview, what we’re sharing today can be considered a narrow distribution model: it's pre-trained on video of the world, and post-trained on video from a smaller set of places with dense coverage. The tradeoff of this post-training is tha…
Could probably be (semi?)automated to run on 3d models of places that doesn't exist. Even ai-built 3d models.
The paper they’re basing this off already does this.
Re: AI video you can watch and interact with, in real-time
#49That felt so wrong AND someone is cheating here. This felt really suspicious... I got to the graffiti world and there were some stairs right next to me. So I started going up them. It felt like I was walking forward and the stairs were pushing under me until I just got stuck. So I turned to go back down and half way around everything morphed and I ended up back down at the ground level where I originally was. I was t…
[deleted]
Re: AI video you can watch and interact with, in real-time
#50That felt so wrong AND someone is cheating here. This felt really suspicious... I got to the graffiti world and there were some stairs right next to me. So I started going up them. It felt like I was walking forward and the stairs were pushing under me until I just got stuck. So I turned to go back down and half way around everything morphed and I ended up back down at the ground level where I originally was. I was t…
Hi! CEO of Odyssey here. Thanks for giving this a shot.
To clarify: this is a diffusion model trained on lots of video, that's learning realistic pixels and actions. This model takes in the prior video frame and a user action (e.g. move forward), with the model then generating a new video frame that resembles the intended action. This loop happens every ~40ms, so real-time.
The reason you're seeing similar worlds with this production model is that one of the greatest challenges of world models is maintaining coherence of video over long time periods, especially with diverse pixels (i.e. not a single game). So, to increase reliability for this research preview—meaning multiple minutes of coherent video—we post-trained this model on video from a smaller set of places with dense coverage. With this, we lose generality, but increase coherence.
We share a lot more about this in our blog post here (https://odyssey.world/introducing-interactive-video), and share outputs from a more generalized model.
> One of the biggest challenges is that world models require autoregressive modeling, predicting future state based on previous state. This means the generated outputs are fed back into the context of the model. In language, this is less of an issue due to its more bounded state space. But in world models—with a far higher-dimensional state—it can lead to instability, as the model drifts outside the support of its training distribution. This is particularly true of real-time models, which have less capacity to model complex latent dynamics.
> To improve autoregressive stability for this research preview, what we’re sharing today can be considered a narrow distribution model: it's pre-trained on video of the world, and post-trained on video from a smaller set of places with dense coverage. The tradeoff of this post-training is that we lose some generality, but gain more stable, long-running autoregressive generation.
> To broaden generalization, we’re already making fast progress on our next-generation world model. That model—shown in raw outputs below—is already demonstrating a richer range of pixels, dynamics, and actions, with noticeably stronger generalization.
Let me know any questions. Happy to go deeper!