Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Worldsheet: Wrapping the World in a 3D Sheet, View Synthesis from a Single Image

worldsheet.github.io

Post reply on HN
Print view View original

21–30 of 34 posts

Re: Worldsheet: Wrapping the World in a 3D Sheet, View Synthesis from a Single Image

#21

Post by andybak »

taneq wrote: post #18

Does anyone know what the Facebook "3D Photos" feature is doing and how/if it's different to this? Because that is generally terrible (much worse than the 'failure mode' examples here) and the depth map generated seems largely arbitrary.

It's worth making a distinction between "3D Photos where the user supplies depth data from the dual cameras/LIDAR etc" vs "3D Photos where Facebook's magic algorithms try to infer depth". It's the latter that I presume you're criticising here. (Might be obvious to you but I thought it worth spelling out)

Re: Worldsheet: Wrapping the World in a 3D Sheet, View Synthesis from a Single Image

#22

Post by yorwba »

moron4hire wrote: post #11
mrfusion wrote: post #10

In the future you could train a model to predict the correct mesh for a given image. Then things get interesting!

That's what they are doing. The rest of the process is not novel.

They're not trying to get the correct mesh, they just treat everything as part of one big continuous surface.

That's good enough for sideways views, but fails if you try to look behind objects, because in their approach, there's no "behind", just a big, stretchy sheet covering everything.

A model with more knowledge about the world would be able to predict that a tree trunk is roughly cylindrical and not connected to the background.

Re: Worldsheet: Wrapping the World in a 3D Sheet, View Synthesis from a Single Image

#23

Post by ben_w »

mrfusion wrote: post #9

Could this be a camera app on the latest iPhones with the built in lidar like technology? You’d snap an image and a mesh, right?

You could, but you don’t need to use a modern phone with a depth sensor – this works with pure RGB and doesn’t need depth information.

Re: Worldsheet: Wrapping the World in a 3D Sheet, View Synthesis from a Single Image

#24

Post by ben_w »

andybak wrote: post #21
taneq wrote: post #18

Does anyone know what the Facebook "3D Photos" feature is doing and how/if it's different to this? Because that is generally terrible (much worse than the 'failure mode' examples here) and the depth map generated seems largely arbitrary.

It's worth making a distinction between "3D Photos where the user supplies depth data from the dual cameras/LIDAR etc" vs "3D Photos where Facebook's magic algorithms try to infer depth". It's the latter that I presume you're criticising here. (Might be obvious to you but I thought it worth spelling out)

I think this paper is also about the latter, it’s just a much better version of the latter - the abstract says RGB image not RGBZ, and one of the examples is a painting.

Re: Worldsheet: Wrapping the World in a 3D Sheet, View Synthesis from a Single Image

#25

Post by steerablesafe »

btown wrote: post #14

To quote the paper: > Our model is supervised with paired input and target views of a scene (along with their camera poses)... The model then needs just a single image at test time. Correct me if I’m wrong, but: Given a novel scene, it seems the model must be retrained on multiple images of that scene? It seems disingenuous, then, to say it works from a single image. No doubt the interpolation is state of the art, bu…

I don't see how that would work with the paintings.

Re: Worldsheet: Wrapping the World in a 3D Sheet, View Synthesis from a Single Image

#26

Post by anorakoverflow »

nayuki wrote: post #15

It'd be nice to see this new technique replace the very crude warping animation in Google maps' street view.

I’ve tried out Apple’s Look Around recently and was impressed with the transition animation compared to Google Maps. If you have an Apple device, try it out in London for instance: Moving past a red telephone box shows how the object kind of retains its shape during the transition. Traffic moving around you also looks quite different compared to Google.

Re: Worldsheet: Wrapping the World in a 3D Sheet, View Synthesis from a Single Image

#27

Post by jack_riminton »

yorwba wrote: post #22
moron4hire wrote: post #11

Earlier quoted context omitted.

That's what they are doing. The rest of the process is not novel.

They're not trying to get the correct mesh, they just treat everything as part of one big continuous surface. That's good enough for sideways views, but fails if you try to look behind objects, because in their approach, there's no "behind", just a big, stretchy sheet covering everything. A model with more knowledge about the world would be able to predict that a tree trunk is roughly cylindrical and not connected to…

I think this is what they were referring to when they mentioned GANs

This paper does a good job of guessing what should go in a scene when given depths and shapes https://www.youtube.com/watch?v=u4HpryLU-VI&ab_channel=TwoMi...

A combination of the two would be wild

Re: Worldsheet: Wrapping the World in a 3D Sheet, View Synthesis from a Single Image

#28

Post by taneq »

andybak wrote: post #21
taneq wrote: post #18

Does anyone know what the Facebook "3D Photos" feature is doing and how/if it's different to this? Because that is generally terrible (much worse than the 'failure mode' examples here) and the depth map generated seems largely arbitrary.

It's worth making a distinction between "3D Photos where the user supplies depth data from the dual cameras/LIDAR etc" vs "3D Photos where Facebook's magic algorithms try to infer depth". It's the latter that I presume you're criticising here. (Might be obvious to you but I thought it worth spelling out)

True, and the latter is indeed what I was referring to. This looks like it does the same thing as Facebook's (train a neural net to infer depth from image contents) but it actually works fairly well.

Generating new views from an rgb+depth image is relatively straightforward and I'd expect any reasonable implementation to work pretty well.

Re: Worldsheet: Wrapping the World in a 3D Sheet, View Synthesis from a Single Image

#29

Post by svantana »

btown wrote: post #14

To quote the paper: > Our model is supervised with paired input and target views of a scene (along with their camera poses)... The model then needs just a single image at test time. Correct me if I’m wrong, but: Given a novel scene, it seems the model must be retrained on multiple images of that scene? It seems disingenuous, then, to say it works from a single image. No doubt the interpolation is state of the art, bu…

I think you're misunderstanding the paper. They are using multiple images in the learning phase to infer the 3d structure. At test time, it works on novel scenes with just a single image.

Re: Worldsheet: Wrapping the World in a 3D Sheet, View Synthesis from a Single Image

#30

Post by moron4hire »

crazygringo wrote: post #16
nayuki wrote: post #15

It'd be nice to see this new technique replace the very crude warping animation in Google maps' street view.

Google Maps Street View already collects some low-res lidar data, so they don't even need this. They just need to build it into their viewer, which I also wish they would do.

They do use the Lidar data in the viewer. You can see it with the positioning of the cursor on "surfaces" when you're looking around. The reprojection animation they do between photospheres is based on the Lidar data, it's just that's about the best you can do with the low-res data they've collected.

Imagine a situation where you're standing on a street corner with a mailbox or something standing on the corner. You move to the next photo position, which takes you past the mailbox. The smearing and warping you are seeing is from the continous mesh they've created from the Lidar data not matching the real world mesh in areas that were occluded from the original point of view. You get a moment of seeing "behind" the mailbox, but there is no data for what is behind the mailbox, so it all gets interpolated from the data that is known in the surrounding visual area. The mailbox becomes a rectangular prism that extends all the way from the mailbox's location through to the intersection with the ground that we can see behind the mailbox.

These are just the problems with single-image mesh recreation. You can't really get around them without some form of inference of the data that doesn't exist. You even see it in Facebook's images in the linked article, if you look closely. They try to cut the videos early so you don't see it, but it's there if you know what to look for.

Post reply on HN