Worldsheet: Wrapping the World in a 3D Sheet, View Synthesis from a Single Image • Hacker News Board

Re: Worldsheet: Wrapping the World in a 3D Sheet, View Synthesis from a Single Image

#31

Post by btown » Mon, Jan 04, 2021, 6:06 PM UTC

svantana wrote: post #29

btown wrote: post #14

To quote the paper: > Our model is supervised with paired input and target views of a scene (along with their camera poses)... The model then needs just a single image at test time. Correct me if I’m wrong, but: Given a novel scene, it seems the model must be retrained on multiple images of that scene? It seems disingenuous, then, to say it works from a single image. No doubt the interpolation is state of the art, bu…

I think you're misunderstanding the paper. They are using multiple images in the learning phase to infer the 3d structure. At test time, it works on novel scenes with just a single image.

I believe from a second reading this is correct - after training on the Matterport real estate image dataset, it appears they can transform novel images (which happen to be from that dataset, so they are still home interiors but can handle completely distinct scenes with distinct furniture etc.). Which is actually really fascinating and leverages decades of innovation. This is actually really cool and fully justifies the hype in the title.

Re: Worldsheet: Wrapping the World in a 3D Sheet, View Synthesis from a Single Image

#32

Post by moetsi » Mon, Jan 04, 2021, 8:21 PM UTC

andybak wrote: post #21

taneq wrote: post #18

Does anyone know what the Facebook "3D Photos" feature is doing and how/if it's different to this? Because that is generally terrible (much worse than the 'failure mode' examples here) and the depth map generated seems largely arbitrary.

It's worth making a distinction between "3D Photos where the user supplies depth data from the dual cameras/LIDAR etc" vs "3D Photos where Facebook's magic algorithms try to infer depth". It's the latter that I presume you're criticising here. (Might be obvious to you but I thought it worth spelling out)

the latter was 6d.ai's whole thing. I say "was" because they were bought by Niantic

Re: Worldsheet: Wrapping the World in a 3D Sheet, View Synthesis from a Single Image

#33

Post by crazygringo » Mon, Jan 04, 2021, 11:03 PM UTC

moron4hire wrote: post #30

crazygringo wrote: post #16

Earlier quoted context omitted.

Google Maps Street View already collects some low-res lidar data, so they don't even need this. They just need to build it into their viewer, which I also wish they would do.

They do use the Lidar data in the viewer. You can see it with the positioning of the cursor on "surfaces" when you're looking around. The reprojection animation they do between photospheres is based on the Lidar data, it's just that's about the best you can do with the low-res data they've collected. Imagine a situation where you're standing on a street corner with a mailbox or something standing on the corner. You m…

> They do use the Lidar data in the viewer.

Oh wow, you're right, I just tried it out. But it's so blocky and apparently limited to (mostly) 90° geometry, that I'd never realized Google was doing anything but modeling a one-size-fits-all "rectangular corridor" along each street.

It seems like it's not exposing any kind of raw Lidar data, but a very simple geometric simplification of it. (E.g. trees are either ignored completely, or if there are enough of them they're treated as the side of a building instead.)

I completely understand what you're saying about the problems with smearing and warping due to not enough data. But I still can't help but wonder what it would look like if they were able to generate a "raw" (but denoised) Lidar geometry, so that trees and cars were treated as individual objects, rather than just either as part of the street floor or part of the building walls they way they are now.

Re: Worldsheet: Wrapping the World in a 3D Sheet, View Synthesis from a Single Image

#34

Post by moron4hire » Tue, Jan 05, 2021, 11:06 AM UTC

sitkack wrote: post #12

moron4hire wrote: post #6

I wonder how long it takes to generate these images. It's not terribly hard to do by hand with rudimentary 3D modeling skills (i.e the most generous way to describe my own 3D modeling capability), if you know the height from the floor and shooting angle at which the camera took the picture. I've done it with 360 photos (therefore shooting angle didn't matter) in Blender with a VR headset to aid in positioning vertice…

15 minutes. I dont know about this particular technique, but others are in the 10-50ms range. Roughly 20k times faster.

My point is, if you need this sort of imagery, you don't need to wait for Facebook to never actually release a usable product out of this paper. Because that's generally what happens with these kinds of papers.