Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia Research Turns 2D Photos into 3D Scenes

blogs.nvidia.com

Post reply on HN
Print view View original

101–110 of 239 posts

Re: Nvidia Research Turns 2D Photos into 3D Scenes

#101

Post by spacexsucks »

ineedasername wrote: post #62
olladecarne wrote: post #46

Earlier quoted context omitted.

There's so many things we invent with good intentions but in the end go terribly wrong and I think this is one of those things. I think it's ok to mourn and remember the past, but moving on and accepting reality is important to a healthy life. Let's be real though, the startup that makes this but appeals to our worst instincts make bank. I can't imagine how much more messed up future generations will be as we keep ma…

"mourn and move on" is a somewhat Western concept of dealing with death. Plenty of cultures around the world have developed different practices, up to religious forms of ancestor worship.

Bs. People do different things. But nobody can sit around wallowing in the VR. Having seen what death of my cousin and grand parents did to the family, moving on helped every one heal in a way. VR would have been a torture to live in.

Re: Nvidia Research Turns 2D Photos into 3D Scenes

#102

Post by hwers »

SV_BubbleTime wrote: post #96
jowday wrote: post #16

Earlier quoted context omitted.

This sort of stuff (generating 3D assets from photographs of real objects) has been common for quite a while via photogrammetry. NeRFs are interesting because (in some cases) they can create renders that look higher quality with fewer photos, and they hint at the potential of future learned rendering models.

I did it for a project… it was SLOW! Gave decent rules for a large area, but poor results for small details. The real trick is textures. When the photo is laid over and wrapped on the model, it looks great. But once you remove that, the raw mesh underneath is not as impressive. I would really like to see the examples they had here without the texture laid on top.

Given the same amount of compute as this used, photogrammetry would be about as fast.

Re: Nvidia Research Turns 2D Photos into 3D Scenes

#103

Post by wokwokwok »

p1esk wrote: post #59
sorenjan wrote: post #44

I don't really understand why NeRFs would be particularly useful in more than a few niche cases, perhaps because I don't fully understand what they really are. My impression is that you take a bunch of photos in various places and directions, then you use those as samples of a 3D function that describes the full scene, and optimize a neural network to minimize the difference between the true light field and what's de…

What happens if you want to animate part of a NeRF, or crop it, or change it in any way? Do you have to throw away all trained coefficients and start again from training data? You don’t change NeRF (the model). You change the point of view of an observer.

I mean, this is the parent posts point; the use case for a static photo or a static 3d nerf is pretty limited.

With other structured data compositing and animating is relatively trivial.

It turns out that people have approached this problem before and you can composite nerf too (1) by sampling different functions over the volume.

…but, let’s not pretend.

The complaint is entirely valid. You’re taking a high resolution voxel grid and encoding it into a model.

Working with simple voxel data let’s you do all kinds of normal image manipulation techniques, and it’s not clear how you would do some of those with a nerf.

Practically speaking, the applications you can use this for are therefore reasonably limited right now.

[1] - https://www.unite.ai/st-nerf-compositing-and-editing-for-vid...

Re: Nvidia Research Turns 2D Photos into 3D Scenes

#104

Post by sorenjan »

corysama wrote: post #88
sorenjan wrote: post #44

I don't really understand why NeRFs would be particularly useful in more than a few niche cases, perhaps because I don't fully understand what they really are. My impression is that you take a bunch of photos in various places and directions, then you use those as samples of a 3D function that describes the full scene, and optimize a neural network to minimize the difference between the true light field and what's de…

If you tried to repro these results (including time & space constraints) using traditional photogrammetry, you would be sorely disappointed. Photogrammetry is great if you have a very solid object that is not shiny or translucent at all. You get a lot of surface color micro-detail and a bit of bumpy meso-detail. But, if something is fuzzy, hairy, or lacey or smokey you are straight-up out of luck. Don't even try. If…

Maybe NeRFs can be used as an intermediate step to reconstruct the scene, and then extract the surfaces and their materials to more conventional representations like meshes, textures, refraction index, etc. I guess the main benefit is that it fills in the undersampled areas in a scene, whether that's an occlusion, a reflection angle, or something else.

My main problem with them is that it seems as if all the data is unstructured and interdependent, not like pixels, voxels, or similar where you can clearly extract and manipulate parts of the data and know what it means. To use your photograph example, a digital photo is a simple grid of colored points, and it's easy to change them individually. A regular 3D scene is a collection of well defined vertices, triangles, materials etc, that is then rendered into a digital photo using a easy to describe process. A NeRF on the other hand seems to be more like enter camera pose => magic => inferred image.

Maybe I'm overthinking it and it doesn't have to be as general as out current formats, maybe a binary blob that can represent a static scene is fine for plenty of applications. But it feels needlessly complicated.

Re: Nvidia Research Turns 2D Photos into 3D Scenes

#105

Post by vmception »

anyfactor wrote: post #14

Tangent I wonder what happens to most people when they see innovation such as this. Over the years I have seen numerous mind-blowing AI achievement, which essentially feel like miracles. Yet literally after an hour I forget what I even saw. I don't find these innovations to have a lasting impression on me or on the internet except for the times when these solutions are released to the public for tinkering and they en…

I immediately think of profitable applications

Another comment mentioned education: I take classes, formal education, for things that are immediately applicable for me right at the time

So its pretty consistent for me

Another thing that might be different is that I don't go for perfect or good enough, or worry that an existing alternative like "having an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem" might already exist, I just go for novelty and am selling to people that also like novelty

Re: Nvidia Research Turns 2D Photos into 3D Scenes

#106

Post by hwers »

daenz wrote: post #2

>The model requires just seconds to train on a few dozen still photos — plus data on the camera angles they were taken from — and can then render the resulting 3D scene within tens of milliseconds. Generating the novel viewpoints is almost fast enough for VR, assuming you're tethered to a desktop computer with whatever GPUs they're using (probably the best setup possible). The holy grail (from my estimation) is getti…

You can just make a mesh once you have the NeRF, which is plenty fast for VR. (What's important isn't new perspective generation but scene training. New perspective generation doesn't have to be real time just within a reasonable time to preprocess and then make a mesh.)

Re: Nvidia Research Turns 2D Photos into 3D Scenes

#107

Post by ineedasername »

spacexsucks wrote: post #101
ineedasername wrote: post #62

Earlier quoted context omitted.

"mourn and move on" is a somewhat Western concept of dealing with death. Plenty of cultures around the world have developed different practices, up to religious forms of ancestor worship.

Bs. People do different things. But nobody can sit around wallowing in the VR. Having seen what death of my cousin and grand parents did to the family, moving on helped every one heal in a way. VR would have been a torture to live in.

You are taking the worst possible interpretation of my comment. I was responding to literally the aspect of the GP that said "mourn.... move on"

It is a factual matter, easily discovered through simple searches, that non-Western cultures often take a different approach. Including, literally, ancestor worship. I am not making a judgement on which is better. I believe there are multiple healthy ways of dealing with grief, and "mourn & move on" can be one, but not the only one.

This is very far from an obsessive interaction with a semi-emulated version of a dead person in VR. However things wouldn't have to be that extreme.

In cultures that practice it, it's not uncommon for a small shrine in the home to be setup. And yet that is of limited access to family located further away, so a digital form of this not bound to a specific location could also be of use to people from these cultures. There is no reason that such things couldn't compliment existing practices of honoring ancestors.

Re: Nvidia Research Turns 2D Photos into 3D Scenes

#109

Post by Jach »

chrstphrknwtn wrote: post #60
baron816 wrote: post #31

It would be really great to recreate loved ones after they have past in some sort of digital space. As I’ve gotten older, and my parents get older as well, I’ve been thinking more about what my life will be like in old age (and beyond too). I’ve also been thinking what I would want “heaven” to be. Eternal life doesn’t appeal to me much. Imagine living a quadrillion years. Even as a god, that would be miserable. That…

My gut reaction to 'recreating loved ones in VR' is that it would be torturous more than soothing. Once a person dies, they're gone. The world is different, and nothing will bring them back. Spending time with a simulacrum isn't really spending time with a loved one.

Don't knock it til you try it?

I mean, already in recent years people have made some very low fidelity 'resurrections' and gotten some measure of comfort from it, never mind the many years of history of people who visit gravestones to 'chat' (some even believing they get replies to some extent or another). When markov chains were hip, "talk with Charles Dickens!" (play with a markov model trained on his works) was at least interesting to some, GPT of course can do a lot better. Imagine we had actual superintelligent AI working on this, which actually tried to recreate brain models, or even restore a sense of existence to the resurrection that can continue independently and to their own delight rather than just being some static VR experience turned on and off whenever. (I jotted some thoughts a few years ago... https://www.thejach.com/view/2017/6/how_you_might_see_some_o...)

I'm in agreement though at least for now that even if you dialed up the fidelity to the point I can't point to anything objectively "off", let alone went with a low fidelity VR thing or just a chat bot, I'd still always think in the back of my mind that this isn't really the same person. Nevertheless, it could still be comforting to some, and interesting to others, and so if it's at all possible for potential future humans/ems/aligned AIs to work at it, they will.

Curiously I don't have the same back of the mind feeling at all when considering the idea of someone preserved with cryonics and then brought back as an em, the person would be the same to me, even if there was a bit of damage from the vitrification process that was error-corrected. I have a disagreement with a friend on that point, who thinks it would be something similar but not the same. Anyway, I think this is due to just having a much higher fidelity "source of truth" to work from that is the preserved adult brain, whereas someone supposedly information-theoretically dead requires a lot more guess-work, perhaps truly impossibly too much more even for a superintelligence, to bring back convincingly.

Re: Nvidia Research Turns 2D Photos into 3D Scenes

#110

Post by jayd16 »

jvanderbot wrote: post #82

> Creating a 3D scene with traditional methods takes hours or longer, depending on the complexity and resolution of the visualization. This just isn't true. I can create a 3D scene from 360-degree photos (even 4) in a minute or so using traditional methods, even open-source toolkits. It doesn't look as good as this because it doesn't have a neural net smoothing the gaps, but it's not true that it takes hours to build…

I think it's comparing to crunching through many more images to get a comparable quality scene.
Post reply on HN