Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI video you can watch and interact with, in real-time

experience.odyssey.world

Post reply on HN
Print view View original

51–60 of 91 posts

Re: AI video you can watch and interact with, in real-time

#51

Post by christianqchung »

AnotherGoodName wrote: post #28
godelski wrote: post #27

That felt so wrong AND someone is cheating here. This felt really suspicious... I got to the graffiti world and there were some stairs right next to me. So I started going up them. It felt like I was walking forward and the stairs were pushing under me until I just got stuck. So I turned to go back down and half way around everything morphed and I ended up back down at the ground level where I originally was. I was t…

Yes the thing that got me was i went through the channels multiple times (multiple browser sessions). The channels are the same everytime (the numbers don't align to any navigation though - flip back and forth between two numbers and you'll just hit a random channel everytime - don't be fooled by that). Every object is in the same position and the layout is the same. What makes this AI generated over just rendering a…

Is it possible that this behavior is a result from training on Google Maps or something similar? I tried to walk off a bridge and you get completely stuck, which is the only reason I can think of that, other than not having first person video views of people walking off bridges.

Re: AI video you can watch and interact with, in real-time

#52

Post by echelon »

AnotherGoodName wrote: post #38
AnotherGoodName wrote: post #28

Earlier quoted context omitted.

Yes the thing that got me was i went through the channels multiple times (multiple browser sessions). The channels are the same everytime (the numbers don't align to any navigation though - flip back and forth between two numbers and you'll just hit a random channel everytime - don't be fooled by that). Every object is in the same position and the layout is the same. What makes this AI generated over just rendering a…

Ok playing with this more there's very subtle differences between sessions. As in there is some hallucination here with certain small differences. I think what's happening is this is AI generated but it is very very overfitted to real world 3D scenes. The AI is almost rendering exactly a real world scene and not much more. They can't travel out of bounds or the model stops working since it's so overfitted to these sc…

Odyssey Systems is six months behind way more impressive demos. They're following in the footsteps of this work:

- Open Source Diamond WM that you can run on consumer hardware [1]

- Google's Genie 2 (way better than this) [2]

- Oasis [3]

[1] https://diamond-wm.github.io/

[2] https://deepmind.google/discover/blog/genie-2-a-large-scale-...

[3] https://oasis.decart.ai/welcome

There are a lot of papers and demos in this space. They have the same artifacts.

Re: AI video you can watch and interact with, in real-time

#54

Post by dragonwriter »

vouaobrasil wrote: post #22

I think this step towards a more immerse virtual reality can actually be dangerous. A lot of intellectual types might disagree but I do think that creating such immersion is a dangerous thing because it will reduce the value people place on the real world and especially the natural world, making them even less likely to care if big corporations screw it up with biospheric degradation. It seems like it has a high chan…

> I think this step towards a more immerse virtual reality can actually be dangerous

I don't think its a step toward that; I think this is literally trained using techniques to generate more immersive virtual reality that already exists and takes less compute, to produce a more computationally expensive and less accurate AI version.

At least, that's what every other demo of a real-time interactive AI world model has been, and they aren't trumpeting any clear new distinction.

Re: AI video you can watch and interact with, in real-time

#55

Post by ollin »

AnotherGoodName wrote: post #38
AnotherGoodName wrote: post #28

Earlier quoted context omitted.

Yes the thing that got me was i went through the channels multiple times (multiple browser sessions). The channels are the same everytime (the numbers don't align to any navigation though - flip back and forth between two numbers and you'll just hit a random channel everytime - don't be fooled by that). Every object is in the same position and the layout is the same. What makes this AI generated over just rendering a…

Ok playing with this more there's very subtle differences between sessions. As in there is some hallucination here with certain small differences. I think what's happening is this is AI generated but it is very very overfitted to real world 3D scenes. The AI is almost rendering exactly a real world scene and not much more. They can't travel out of bounds or the model stops working since it's so overfitted to these sc…

I think the most likely explanation is that they trained a diffusion WM (like DIAMOND) on video rollouts recorded from within a 3D scene representation (like NeRF/GS), with some collision detection enabled.

This would explain:

1. How collisions / teleportation work and why they're so rigid (the WM is mimicking hand-implemented scene-bounds logic)

2. Why the scenes are static and, in the case of should-be-dynamic elements like water/people/candles, blurred (the WM is mimicking artifacts from the 3D representation)

3. Why they are confident that "There's no map or explicit 3D representation in the outputs. This is a diffusion model, and video in/out" https://x.com/olivercameron/status/1927852361579647398 (the final product is indeed a diffusion WM trained on videos, they just have a complicated pipeline for getting those training videos)

Re: AI video you can watch and interact with, in real-time

#56

Post by olivercameron »

abe94 wrote: post #41

very cool - what was the hardest part of building this?

If I had to choose one, I'd easily say maintaining video coherence over long periods of time. The typical failure case of world models that's attempting to generate diverse pixels (i.e. beyond a single video game) is that they degrade to a mush of incoherent pixels after 10-20 seconds of video.

We talk about this challenge in our blog post here (https://odyssey.world/introducing-interactive-video). There's specifics in there on how we improved coherence for this production model, and our work to improve this further with our next-gen model. I'm really proud of our work here!

> Compared to language, image, or video models, world models are still nascent—especially those that run in real-time. One of the biggest challenges is that world models require autoregressive modeling, predicting future state based on previous state. This means the generated outputs are fed back into the context of the model. In language, this is less of an issue due to its more bounded state space. But in world models—with a far higher-dimensional state—it can lead to instability, as the model drifts outside the support of its training distribution. This is particularly true of real-time models, which have less capacity to model complex latent dynamics. Improving this is an area of research we're deeply invested in.

In second place would absolutely be model optimization to hit real-time. That's a gnarly problem, where you're delicately balancing model intelligence, resolution, and frame-rate.

Re: AI video you can watch and interact with, in real-time

#59

Post by jowday »

olivercameron wrote: post #50
godelski wrote: post #27

That felt so wrong AND someone is cheating here. This felt really suspicious... I got to the graffiti world and there were some stairs right next to me. So I started going up them. It felt like I was walking forward and the stairs were pushing under me until I just got stuck. So I turned to go back down and half way around everything morphed and I ended up back down at the ground level where I originally was. I was t…

Hi! CEO of Odyssey here. Thanks for giving this a shot. To clarify: this is a diffusion model trained on lots of video, that's learning realistic pixels and actions. This model takes in the prior video frame and a user action (e.g. move forward), with the model then generating a new video frame that resembles the intended action. This loop happens every ~40ms, so real-time. The reason you're seeing similar worlds wit…

Why are you going all in on world models instead of basing everything on top of a 3D engine that could be manipulated / rendered with separate models? If a world model was truly managing to model a manifold of a 3D scene, it should be pretty easy to extract a mesh or SDF from it and drop that into an engine where you could then impose more concrete rules or sanity check the output of the model. Then you could actually model player movement inside of the 3D engine instead of trying to train the world model to accept any kind of player input you might want to do now or in the future.

Additionally, curious about what exactly the difference between the new mode of storytelling you’re describing and something like a crpg or visual novel is - is your hope that you can just bake absolutely everything into the world model instead of having to implement systems for dialogue/camera controls/rendering/everything else that’s difficult about working with a 3D engine?

Re: AI video you can watch and interact with, in real-time

#60

Post by Animats »

godelski wrote: post #27

That felt so wrong AND someone is cheating here. This felt really suspicious... I got to the graffiti world and there were some stairs right next to me. So I started going up them. It felt like I was walking forward and the stairs were pushing under me until I just got stuck. So I turned to go back down and half way around everything morphed and I ended up back down at the ground level where I originally was. I was t…

> Not unless the model is building mini worlds with boundaries.

Right. I was never able to get very far from the starting point, and kept getting thrown back to the start. It looks like they generated a little spherical image, and they're able to extrapolate a bit from that. Try to go through a door or reach a distant building, and you don't get there.

Post reply on HN