Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia Research Turns 2D Photos into 3D Scenes

blogs.nvidia.com

Post reply on HN
Print view View original

11–20 of 239 posts

Re: Nvidia Research Turns 2D Photos into 3D Scenes

#11

Post by krasin »

c4wrd wrote: post #10
simsla wrote: post #5

Earlier quoted context omitted.

> plus data on the camera angles they were taken from Doesn't seem like much of a stretch to determine the angles as well. E.g. a semi brute forced way with GANs

I've spent a lot of time thinking about this (i.e. taking a video and creating a 3D scene) and I don't think that it is feasible in most cases to have good accuracy. If you need to infer the angle, you need make a lot of biased assumptions about things like velocity, position, etc., of the camera and even if you were 99.9% accurate, that 0.1% inaccuracy is compounded over time. Now I'm not saying it's not possible, b…

https://github.com/NVLabs/instant-ngp has a script that converts a video into frames and then uses COLMAP ([1]) to compute camera poses. You can then train a NeRF model within a few seconds.

It all works pretty well. Trying it on your own video is pretty straightforward.

1. https://colmap.github.io/

Re: Nvidia Research Turns 2D Photos into 3D Scenes

#12

Post by riotnrrd »

simsla wrote: post #5
daenz wrote: post #2

>The model requires just seconds to train on a few dozen still photos — plus data on the camera angles they were taken from — and can then render the resulting 3D scene within tens of milliseconds. Generating the novel viewpoints is almost fast enough for VR, assuming you're tethered to a desktop computer with whatever GPUs they're using (probably the best setup possible). The holy grail (from my estimation) is getti…

> plus data on the camera angles they were taken from Doesn't seem like much of a stretch to determine the angles as well. E.g. a semi brute forced way with GANs

You don't even need anything that fancy. Traditional structure-from-motion, or visual odometry gives accurate enough position estimations.

If you want to experiment, take a bunch (~100) of photos of an object, and use COLMAP to generate the poses. COLMAP implements a global SfM technique, so it will be very accurate but very slow.

Re: Nvidia Research Turns 2D Photos into 3D Scenes

#13

Post by riotnrrd »

c4wrd wrote: post #10
simsla wrote: post #5

Earlier quoted context omitted.

> plus data on the camera angles they were taken from Doesn't seem like much of a stretch to determine the angles as well. E.g. a semi brute forced way with GANs

I've spent a lot of time thinking about this (i.e. taking a video and creating a 3D scene) and I don't think that it is feasible in most cases to have good accuracy. If you need to infer the angle, you need make a lot of biased assumptions about things like velocity, position, etc., of the camera and even if you were 99.9% accurate, that 0.1% inaccuracy is compounded over time. Now I'm not saying it's not possible, b…

You're far too pessimistic (or maybe you don't know the field well). The problem of estimating the relative poses of the cameras responsible for a set of photos is a long standing and essentially "solved" problem in computer vision. I say "solved" because there is still active research (increasing accuracy, faster, more robust, etc.) but there are decades-old, well known techniques that any dedicated programmer could implement in a week.

If you're genuinely curious, look into structure from motion, visual odometry, or SLAM.

Re: Nvidia Research Turns 2D Photos into 3D Scenes

#14

Post by anyfactor »

Tangent

I wonder what happens to most people when they see innovation such as this. Over the years I have seen numerous mind-blowing AI achievement, which essentially feel like miracles. Yet literally after an hour I forget what I even saw. I don't find these innovations to have a lasting impression on me or on the internet except for the times when these solutions are released to the public for tinkering and they end up failing catastrophically.

I remember having the same feeling about chatbots and TTS technology literally ages ago, but at present time, the practical use of these innovation feel very mediocre.

Re: Nvidia Research Turns 2D Photos into 3D Scenes

#16

Post by jowday »

elil17 wrote: post #8

My prediction/hope is that NeRFs will totally revolutionize how the film/TV industry. I can imagine: - Shooting a movie from a few cameras, creating a movie version of a NeRF using those angles, and then dynamically adding in other shots in post - Using lighting and depth information embedded in NeRFs to assist in lighting/integrating CG elements - Using NeRFs to generate virtual sets on LED walls (like those on The…

This sort of stuff (generating 3D assets from photographs of real objects) has been common for quite a while via photogrammetry. NeRFs are interesting because (in some cases) they can create renders that look higher quality with fewer photos, and they hint at the potential of future learned rendering models.

Re: Nvidia Research Turns 2D Photos into 3D Scenes

#20

Post by syspec »

Is there a video of this? I'm not sure what's the connection to the top photo/video/matrix-360-effect

Was that created from a few photos? I didn't see any additional imagery below

--- Update

It looks like these are the four source photos: https://blogs.nvidia.com/wp-content/uploads/2022/03/NVIDIA-R...

Then it creates this 360 video from them: https://blogs.nvidia.com/wp-content/uploads/2022/03/2141864_...

Post reply on HN