Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SHARP, an approach to photorealistic view synthesis from a single image

apple.github.io

Post reply on HN
Print view View original

101–110 of 114 posts

Re: SHARP, an approach to photorealistic view synthesis from a single image

#101

Post by alexgotoi »

Apple dropping this is interesting. They've been quiet on the flashy AI stuff while everyone else is yelling about transformers, but 3D reconstruction from single images is actually useful hardware integration stuff.

What's weird is we're getting better at faking 3D from 2D than we are at just... capturing actual 3D data. Like we have LiDAR in phones already, but it's easier to neural-net your way around it than deal with the sensor data properly.

Five years from now we'll probably look back at this as the moment spatial computing stopped being about hardware and became mostly inference. Not sure if that's good or bad tbh.

Will include this one in my https://hackernewsai.com/ newsletter.

Re: SHARP, an approach to photorealistic view synthesis from a single image

#102

Post by [deleted] »

rcarmo wrote: post #94
7moritz7 wrote: post #87

Earlier quoted context omitted.

The example doesn't look particularly impressive to say the least. Look at the bottom 20 %

I just refactored the rendering and resampling approach. Took me a few tries to figure out how to remove the banding masks from the layers, but with more stacked layers and a bit of GPT-foo to figure out the API it sort of works now (updated the GIF) Keep in mind that this is not Gaussian splat rendering but just a hacked approximation--on my NVIDIA machine that looks way smoother.

[deleted]

Re: SHARP, an approach to photorealistic view synthesis from a single image

#104

Post by crazygringo »

emsign wrote: post #34
Leptonmaniac wrote: post #27

Can someone ELI5 what this does? I read the abstract and tried to find differences in the provided examples, but I don't understand (and don't see) what the "photorealistic" part is.

Imagine history documentaries where they take an old photo and free objects from the background and move them round giving the illusion of parallax movement. This software does that in less than a second, creating a 3D model that can be accurately moved (or the camera for that matter) in your video editor. It's not new, but this one is fast and "sharp". Gaussian splashing is pretty awesome.

Oh man. I never thought about how Ken Burns might use that.

Already you sometimes see where manually cut out a foreground person from the background and enlarge them a little bit and create a multi-layer 3D effect, but it's super-primitive and I find it gimmicky.

Bringing actual 3D to old photographs as the camera slowly pans or rotates slightly feels like it could be done really tastefully and well.

Re: SHARP, an approach to photorealistic view synthesis from a single image

#106

Post by momojo »

alexgotoi wrote: post #101

Apple dropping this is interesting. They've been quiet on the flashy AI stuff while everyone else is yelling about transformers, but 3D reconstruction from single images is actually useful hardware integration stuff. What's weird is we're getting better at faking 3D from 2D than we are at just... capturing actual 3D data. Like we have LiDAR in phones already, but it's easier to neural-net your way around it than deal…

I wonder if humans are any different. We don't have LIDAR in our eyes but we approximate depth "enough" with only our 2D input

Re: SHARP, an approach to photorealistic view synthesis from a single image

#108

Post by Someone »

sorenjan wrote: post #96
supermatt wrote: post #78

Earlier quoted context omitted.

Im not sure how the depth estimation alone translates into the view synthesis, but the current implementation on-device is definitely not convincing for literally any portrait photographs I have seen. True stereoscopic captures are convincing statically, but don't provide the parallax.

Good monocular depth estimation is crucial if you want to make a 3D representation from a single image. Ordinarily you have images from several camera poses and can create the gaussian splats using triangulation, with a single image you have to guess z position for them.

For selfies, I think iPhones with Face ID use the TrueDepth camera hardware to measure Z position. That’s not full camera resolution, but it will definitely help.

Re: SHARP, an approach to photorealistic view synthesis from a single image

#110

Post by arjie »

necovek wrote: post #65
arjie wrote: post #8

This is incredibly cool. It's interesting how it fails in the section where you need to in-paint. SVC seems to do that better than all the rest, though not anywhere close to the photorealism of this model. Is there a similar flow but to transform either a video/photo/NeRF of a scene into a tighter, minimal polygon approximation of it. The reason I ask is that it would make some things really cool. To make my baby mon…

You'd still need one real measurement at least: this might get proportions right if background can be clearly separated, but the absolute size of an object can be worlds apart.

That's true. And there's lens correction and all that, but it would be nice to accelerate the CAD modeling.
Post reply on HN