Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia Research Turns 2D Photos into 3D Scenes

blogs.nvidia.com

Post reply on HN
Print view View original

181–190 of 239 posts

Re: Nvidia Research Turns 2D Photos into 3D Scenes

#181

Post by lolspace »

Jach wrote: post #120
lolspace wrote: post #112

Earlier quoted context omitted.

Have you lost someone you've loved?

Yes, but I get the unstated implication. I don't think it would be fair to apply it to me even though I think it may be fair to apply it to e.g. Kurzweil, unless he's made recent statements suggesting otherwise (I don't keep up with him). I currently have no expectation of seeing a convincing simulation/resurrection/recreation/continuation of any of them, or the ones I currently anticipate losing over the next few de…

Would you want to have an avatar of them? I don't think I would.

Re: Nvidia Research Turns 2D Photos into 3D Scenes

#182

Post by dharma1 »

Synaesthesia wrote: post #179
dharma1 wrote: post #178

In terms of practical use - is there a pipeline to use the NeRF 3D scenes in Unreal Engine? How many photos do you need on average vs photogrammetry? 50% less?

This is not a textured polygon asset, it's a neural field, so it's like stored the directional data in a neural network, I think using spherical harmonics.

That’s ok, just need some level of integration with UE (being able to /integrate with UE camera etc). Specifically interested in using this for LED green screens that sync to a live camera movement (Mandalorian style). Our pipeline uses UE, Nvidia cards and photogrammetry/plates/3D models atm, this could speed things up a lot and require less photos for creating the 3D backgrounds.

Re: Nvidia Research Turns 2D Photos into 3D Scenes

#184

Post by Mandelmus »

CobrastanJorji wrote: post #157
corysama wrote: post #88

Earlier quoted context omitted.

If you tried to repro these results (including time & space constraints) using traditional photogrammetry, you would be sorely disappointed. Photogrammetry is great if you have a very solid object that is not shiny or translucent at all. You get a lot of surface color micro-detail and a bit of bumpy meso-detail. But, if something is fuzzy, hairy, or lacey or smokey you are straight-up out of luck. Don't even try. If…

That's more interesting than I realized. In this example, I assumed that the model was generating some sort of 3D mesh representing the woman. Is that not at all the case? Would this technique be unable to generate a model or volumetric information despite being able to reasonably render her from many directions?

No, there is no mesh. A NeRF is a neural network trained to work as a function f(x, y, z, θ, φ). You put in your viewing position (x, y, z) in 3D space and the direction (θ, φ) you're looking into (where θ and φ are the angles for up/down and left/right, respectively), and the function will output a tuple (r, g, b, σ) of the colour (r, g, b) and the material density (σ) of whatever you see at the pixel in that direction and from that position.

You can generate a mesh from the density information this function gives you, but for that you need to discretise the continuous densities you get out.

Re: Nvidia Research Turns 2D Photos into 3D Scenes

#185

Post by Mandelmus »

m3at wrote: post #146

This is great, and the paper+codebase they're referring to (but not linking, here [1]) is neat too. The research is moving fast though, so if you want something almost as fast without specialized CUDA kernels (just plain pytorch) you're in luck: https://github.com/apchenstu/TensoRF As a bonus you also get a more compact representation of the scene. [1] https://github.com/NVlabs/instant-ngp

The key difference is that this TensoRF is discrete (like a JPG) whereas NeRFs are continuous (like vector graphics).

Re: Nvidia Research Turns 2D Photos into 3D Scenes

#186

Post by dijonman2 »

chrstphrknwtn wrote: post #60
baron816 wrote: post #31

It would be really great to recreate loved ones after they have past in some sort of digital space. As I’ve gotten older, and my parents get older as well, I’ve been thinking more about what my life will be like in old age (and beyond too). I’ve also been thinking what I would want “heaven” to be. Eternal life doesn’t appeal to me much. Imagine living a quadrillion years. Even as a god, that would be miserable. That…

My gut reaction to 'recreating loved ones in VR' is that it would be torturous more than soothing. Once a person dies, they're gone. The world is different, and nothing will bring them back. Spending time with a simulacrum isn't really spending time with a loved one.

It has the potential to be therapeutic. Perhaps only to be used under the supervision of a mental health professional.

I for one have many things I wish I had said to my parents before they died. I’d like to use it, but not be obsessed or consumed by it.

Re: Nvidia Research Turns 2D Photos into 3D Scenes

#187

Post by Groxx »

fleischhauf wrote: post #27
anyfactor wrote: post #14

Tangent I wonder what happens to most people when they see innovation such as this. Over the years I have seen numerous mind-blowing AI achievement, which essentially feel like miracles. Yet literally after an hour I forget what I even saw. I don't find these innovations to have a lasting impression on me or on the internet except for the times when these solutions are released to the public for tinkering and they en…

I have the impression, that now some of them seem to really end up in some practical applications. Funnily enough someone just today showed me a feature of his phone where you can select some undesired objects in youe photo and it would just replace them with a fitting background indistinguishable from the original photo.

It's rather entertaining when this happens in the opposite direction automatically too: https://twitter.com/mitchcohen/status/1476351601862483968

Re: Nvidia Research Turns 2D Photos into 3D Scenes

#188

Post by soylentgraham »

hansworst wrote: post #174
SV_BubbleTime wrote: post #96

Earlier quoted context omitted.

I did it for a project… it was SLOW! Gave decent rules for a large area, but poor results for small details. The real trick is textures. When the photo is laid over and wrapped on the model, it looks great. But once you remove that, the raw mesh underneath is not as impressive. I would really like to see the examples they had here without the texture laid on top.

There is no mesh here. Nerfs are 5d (colours are computed based on a 3D position vector + a view direction vector) fields that are rendered volumetrically. So the “texture” is an integral part of the neural representation of the scene, not just an image applied to a mesh. The cool part is that this also allows for capturing transparency, and any effects caused by lighting (including complex specular reflections) are…

Nitpicking, but for GP; Nerf is the internal representation, but the output doesnt have to be 2D (ray traced basically)

There are examples of people outputting SDF (and by extension geometry) with nerf, and projecting original texture onto that would give some nice effects; (live volumetric works best this way) though there would be some disparity where edges/occlusion isnt perfect, so youd want to sample nerf's rgb anyway... although a lot of that is fuzzy at the edges too. A lot of incorrect transparency at edges looks great in the 2D renders (so much anti aliasing and noise!) but less good for texturing

Re: Nvidia Research Turns 2D Photos into 3D Scenes

#189

Post by Razengan »

baron816 wrote: post #31

It would be really great to recreate loved ones after they have past in some sort of digital space. As I’ve gotten older, and my parents get older as well, I’ve been thinking more about what my life will be like in old age (and beyond too). I’ve also been thinking what I would want “heaven” to be. Eternal life doesn’t appeal to me much. Imagine living a quadrillion years. Even as a god, that would be miserable. That…

> It would be really great to recreate loved ones after they have passed

That’s probably the basis for most of human religion and philosophy; coping mechanisms.

I’ve always wondered if it might be possible to ”reverse engineer” parents or siblings from one’s own DNA.. at least visually.

Re: Nvidia Research Turns 2D Photos into 3D Scenes

#190

Post by robertlagrant »

baron816 wrote: post #31

It would be really great to recreate loved ones after they have past in some sort of digital space. As I’ve gotten older, and my parents get older as well, I’ve been thinking more about what my life will be like in old age (and beyond too). I’ve also been thinking what I would want “heaven” to be. Eternal life doesn’t appeal to me much. Imagine living a quadrillion years. Even as a god, that would be miserable. That…

I can imagine people would also create their exes and talk to them after they've broken up.
Post reply on HN