Nvidia Research Turns 2D Photos into 3D Scenes
blogs.nvidia.com
Nvidia Research Turns 2D Photos into 3D Scenes
1–10 of 239 posts
Re: Nvidia Research Turns 2D Photos into 3D Scenes
#2>The model requires just seconds to train on a few dozen still photos — plus data on the camera angles they were taken from — and can then render the resulting 3D scene within tens of milliseconds.
Generating the novel viewpoints is almost fast enough for VR, assuming you're tethered to a desktop computer with whatever GPUs they're using (probably the best setup possible).
The holy grail (from my estimation) is getting both the training and the rendering to fit into a VR frame budget. They'll probably achieve it soon with some very clever techniques that only require differential re-training as the scene changes. The result will be a VR experience with live people and objects that feels photorealistic, because it essentially is based on real photos.
Re: Nvidia Research Turns 2D Photos into 3D Scenes
#3I’m probably going to ramp up the number of photos I take in hope that google photos auto applies this tech
Re: Nvidia Research Turns 2D Photos into 3D Scenes
#4Are there examples of this being used on large outdoor spaces?
Re: Nvidia Research Turns 2D Photos into 3D Scenes
#5>The model requires just seconds to train on a few dozen still photos — plus data on the camera angles they were taken from — and can then render the resulting 3D scene within tens of milliseconds. Generating the novel viewpoints is almost fast enough for VR, assuming you're tethered to a desktop computer with whatever GPUs they're using (probably the best setup possible). The holy grail (from my estimation) is getti…
> plus data on the camera angles they were taken from
Doesn't seem like much of a stretch to determine the angles as well.
E.g. a semi brute forced way with GANs
Re: Nvidia Research Turns 2D Photos into 3D Scenes
#6Nvidia is leaving us all behind
Re: Nvidia Research Turns 2D Photos into 3D Scenes
#7Are there examples of this being used on large outdoor spaces?
Yes, Waymo did the whole San Francisco block: https://waymo.com/research/block-nerf/
Re: Nvidia Research Turns 2D Photos into 3D Scenes
#8My prediction/hope is that NeRFs will totally revolutionize how the film/TV industry. I can imagine:
- Shooting a movie from a few cameras, creating a movie version of a NeRF using those angles, and then dynamically adding in other shots in post
- Using lighting and depth information embedded in NeRFs to assist in lighting/integrating CG elements
- Using NeRFs to generate virtual sets on LED walls (like those on The Mandalorian) from just a couple of photos of a location or a couple of renders of a scene (currently, the sets have to be built in a game engine and optimized for real time performance).
Re: Nvidia Research Turns 2D Photos into 3D Scenes
#9My prediction/hope is that NeRFs will totally revolutionize how the film/TV industry. I can imagine: - Shooting a movie from a few cameras, creating a movie version of a NeRF using those angles, and then dynamically adding in other shots in post - Using lighting and depth information embedded in NeRFs to assist in lighting/integrating CG elements - Using NeRFs to generate virtual sets on LED walls (like those on The…
Computer games, VR and AR could also be pretty amazing
uses for this technique too.
Re: Nvidia Research Turns 2D Photos into 3D Scenes
#10>The model requires just seconds to train on a few dozen still photos — plus data on the camera angles they were taken from — and can then render the resulting 3D scene within tens of milliseconds. Generating the novel viewpoints is almost fast enough for VR, assuming you're tethered to a desktop computer with whatever GPUs they're using (probably the best setup possible). The holy grail (from my estimation) is getti…
> plus data on the camera angles they were taken from Doesn't seem like much of a stretch to determine the angles as well. E.g. a semi brute forced way with GANs
I've spent a lot of time thinking about this (i.e. taking a video and creating a 3D scene) and I don't think that it is feasible in most cases to have good accuracy. If you need to infer the angle, you need make a lot of biased assumptions about things like velocity, position, etc., of the camera and even if you were 99.9% accurate, that 0.1% inaccuracy is compounded over time. Now I'm not saying it's not possible, but I'd believe that if you want an accurate 3D scene, you'd rather be spending your computation budget on things other than determining those angles when it can be simply be provided by hardware.