In the future you could train a model to predict the correct mesh for a given image. Then things get interesting!
That's what they are doing. The rest of the process is not novel.
Worldsheet: Wrapping the World in a 3D Sheet, View Synthesis from a Single Image
11–20 of 34 posts
Re: Worldsheet: Wrapping the World in a 3D Sheet, View Synthesis from a Single Image
#12I wonder how long it takes to generate these images. It's not terribly hard to do by hand with rudimentary 3D modeling skills (i.e the most generous way to describe my own 3D modeling capability), if you know the height from the floor and shooting angle at which the camera took the picture. I've done it with 360 photos (therefore shooting angle didn't matter) in Blender with a VR headset to aid in positioning vertice…
15 minutes. I dont know about this particular technique, but others are in the 10-50ms range. Roughly 20k times faster.
Re: Worldsheet: Wrapping the World in a 3D Sheet, View Synthesis from a Single Image
#13Didn't read the paper in detail, but seems just an extension / application for single image depth prediction problem?
Re: Worldsheet: Wrapping the World in a 3D Sheet, View Synthesis from a Single Image
#14To quote the paper:
> Our model is supervised with paired input and target views of a scene (along with their camera poses)... The model then needs just a single image at test time.
Correct me if I’m wrong, but: Given a novel scene, it seems the model must be retrained on multiple images of that scene? It seems disingenuous, then, to say it works from a single image. No doubt the interpolation is state of the art, but the title seems misleadingly magical.
Re: Worldsheet: Wrapping the World in a 3D Sheet, View Synthesis from a Single Image
#15It'd be nice to see this new technique replace the very crude warping animation in Google maps' street view.
Re: Worldsheet: Wrapping the World in a 3D Sheet, View Synthesis from a Single Image
#16It'd be nice to see this new technique replace the very crude warping animation in Google maps' street view.
Google Maps Street View already collects some low-res lidar data, so they don't even need this.
They just need to build it into their viewer, which I also wish they would do.
Re: Worldsheet: Wrapping the World in a 3D Sheet, View Synthesis from a Single Image
#17It'd be nice to see this new technique replace the very crude warping animation in Google maps' street view.
I believe the crude warp is caused by the difficult combination of:
* Need to do animation with very low latency (can't go to the network to collect data for the animation when the user clicks)
* Need to do the animation without much CPU/GPU power in the browser.
* Need to not download much data for the animation ahead of time (browsing panoramas is already very data heavy).
Re: Worldsheet: Wrapping the World in a 3D Sheet, View Synthesis from a Single Image
#18Does anyone know what the Facebook "3D Photos" feature is doing and how/if it's different to this? Because that is generally terrible (much worse than the 'failure mode' examples here) and the depth map generated seems largely arbitrary.
Re: Worldsheet: Wrapping the World in a 3D Sheet, View Synthesis from a Single Image
#19To quote the paper: > Our model is supervised with paired input and target views of a scene (along with their camera poses)... The model then needs just a single image at test time. Correct me if I’m wrong, but: Given a novel scene, it seems the model must be retrained on multiple images of that scene? It seems disingenuous, then, to say it works from a single image. No doubt the interpolation is state of the art, bu…
Very curious to understand this as well. Does that imply that images with similar attributes (like other hotel rooms in one of their examples with slightly different dimensions or furniture) will work without additional training? Or did each example require multiple image training of that scene before a 3d mesh can be computed from arbitrary angle? In the latter case it can still be useful as compared to photogrammetry which would require hundreds of photos to achieve similar results, so maybe this could work with a dozen or less?
Re: Worldsheet: Wrapping the World in a 3D Sheet, View Synthesis from a Single Image
#20To quote the paper: > Our model is supervised with paired input and target views of a scene (along with their camera poses)... The model then needs just a single image at test time. Correct me if I’m wrong, but: Given a novel scene, it seems the model must be retrained on multiple images of that scene? It seems disingenuous, then, to say it works from a single image. No doubt the interpolation is state of the art, bu…
Unfortunately that's most(all) of academia!