Atlas: A World Model for Spatial Intelligence

Yes, one of the main goals of this model was to keep 3D consistency as the camera moves around; it can do this even without relying on an explicit 3D point cloud or Gaussian splat representation of the scene.

It can operate in different modes for different applications.

If you only want to reconstruct the content in the input images without any imagination, Atlas can do sparse 3D reconstruction where it predicts a depth map for each input pixel, without predicting anything occluded or not visible in the input views. This could be used to implement your "fog of war" view.

But sometimes you want the model to stitch together views and imagine a coherent completion; this is a powerful tool for building and authoring generative worlds. For example the 1 minute video we show in the post was built by positioning seven unrelated input images in space and prompting the model to interpolate between them along a hand-specified camera path.

This is incredible. One potential application that I'm thinking about already is the rapid iteration of video-game map blocking. Being able to drop in some 'initial state' configuration and then have it procedurally generate a handful of alternative configurations could make rapid prototyping a significantly quicker experience, especially if you wanted to see what a potential end result could look like.

Furthermore, being able to extract and process world geometry and 3D objects from Atlas could reduce friction in the early stages of indy development, where developer time is stretched thinner.

I'm very excited about AI tooling moving forward if this is a glimpse into the future.

The blog post doesn't seem to mention what strikes me as the most interesting application of a model like this, namely extracting semantic information from its latent space. It mentions robotics applications, but only in the context of generating realistic world models for simulation.

If you have a robot deployed in an environment, generating synthetic views of the environment you're in doesn't have any obvious value. What does have obvious value is the latent knowledge that the model could have used to generate those synthetic views.

For instance, the fact that Atlas is capable of identifying regions of the input images that look like "floors", and smoothly interpolating them and filling in gaps with more floor, suggests that it has a concept of "floor-like walkability" which it's learned from the examples in its training data. And being able to identify the regions of 3D space that correspond to that semantic label would obviously be useful for robot path planning.

There's plenty of literature about e.g. using neural networks to estimate walkable areas from a point cloud. And you could imagine just bolting one of those methods to the front of Atlas, using the synthesized point cloud (instead of traditional photogrammetry or LIDAR) as input. But that seems like it's throwing away a lot of potentially useful semantic information, on top of being needlessly inefficient.

How long does it take to process from input images to model output?