[deleted]
HybridNeRF: Efficient Neural Rendering
31–40 of 53 posts
Re: HybridNeRF: Efficient Neural Rendering
#32One of the paper authors here - happy to answer any questions about the work or chat about neural rendering in general!
Wow, it looks beautiful.
Can regular phones capture the data required? How to get into this, as a hobbyist? I’m interested in the possibilities of scanning coral reefs and other ecological settings.
Re: HybridNeRF: Efficient Neural Rendering
#33Absolute noob question that I'm having a hard time understading:
In practice, why NeRF instead of Gaussian Splatting? I have very limited exposure to either, but a very cursory search on the subject yields a "it depends on the context" answer. What exact context?
Re: HybridNeRF: Efficient Neural Rendering
#34Earlier quoted context omitted.
You are correct; most of these new techniques are using a camera. In my line of work I consider a camera sensor a scanner of sorts, as we do a lot of photogrammetry and “scan” with a 45MP full frame. The inferred 3D from cameras is pretty bad when it comes to accuracy, especially from dimly lit areas or where you dip into a closet or closed space that doesn’t have a good structural tie back to the main space you are…
Both my iPhone and my Apple Vision Pro both have lidar scanners, fwiw. Frankly I’m surprised that I can’t easily make crude 3D models of spaces with a simple app presently. It seems well within the capabilities of the hardware and software.
Those LiDAR sensors on phones and VR headsets are low resolution and mainly used to improve the photos and depth information from the camera. Different objective than mapping a space, which is mainly being disrupted by improvements from the self driving car and ADAS industries
Re: HybridNeRF: Efficient Neural Rendering
#35Earlier quoted context omitted.
Both my iPhone and my Apple Vision Pro both have lidar scanners, fwiw. Frankly I’m surprised that I can’t easily make crude 3D models of spaces with a simple app presently. It seems well within the capabilities of the hardware and software.
Those LiDAR sensors on phones and VR headsets are low resolution and mainly used to improve the photos and depth information from the camera. Different objective than mapping a space, which is mainly being disrupted by improvements from the self driving car and ADAS industries
Magic Room for the AVP does a good enough job. Seems the low resolution issue can be augmented/improved by repeated/closer scans.
Re: HybridNeRF: Efficient Neural Rendering
#36Absolute noob question that I'm having a hard time understading: In practice, why NeRF instead of Gaussian Splatting? I have very limited exposure to either, but a very cursory search on the subject yields a "it depends on the context" answer. What exact context?
It's just a completely different paradigm of rendering and it's not clear which one will be dominant in the future. Gaussian splats are usually dependent on initialisation from point cloud, which makes whole process much more compliacated.
Re: HybridNeRF: Efficient Neural Rendering
#37One of the paper authors here - happy to answer any questions about the work or chat about neural rendering in general!
Wow, it looks beautiful. Can regular phones capture the data required? How to get into this, as a hobbyist? I’m interested in the possibilities of scanning coral reefs and other ecological settings.
One of the datasets we evaluated against in our paper uses a bespoke capture rig (https://github.com/facebookresearch/EyefulTower?tab=readme-o...) but you can definitely train very respectable NeRFs using a phone camera. In my experience it's less about camera resolution and more about getting a good capture - many NeRF methods assume that the scene is static, so minimizing things like lighting changes and transient shadows can make a big difference. If you're interested in getting your feet wet, I highly recommend Nerfstudio (https://docs.nerf.studio)!
Re: HybridNeRF: Efficient Neural Rendering
#38Earlier quoted context omitted.
I work in the rendering and gaming industry and also run a 3D scanning company. I have similarly wished for this capability, especially the destructability part. What you speak of is still pretty far off for several reasons: -No Collision/poor collision on NERFs and GS: to have a proper interactive world, you usually need accurate character collision so that your character or vehicle can move along the floor/ground (…
I think collision detection is solvable. And the scanning process should be no harder than 3D modeling to the same quality level. Probably much easier, honestly. Modeling is labor intensive. I'm not sure why you say "there’s no scanner available that provides both good 3-D information and good photo realistic textures" because these new techniques don't use "scanners", all you need is regular cameras. The 3D informat…
I feel like the lighting part will become "easy" once we're able to greatly simplify the geometry and correlate it across multiple "passes" through the same space at different times.
In other words, if you've got a consistent 3D geometric map of the house with textures, then you can do a pass in the morning with only daylight, midday only daylight, late afternoon only daylight, and then one at night with artificial light.
If you're dealing with textures that map onto identical geometries (and assume no objects move during the day), it seems like it ought to be relatively straightforward to train AI's to produce a flat unlit texture version, especially since you can train them on easily generated raytraced renderings. There might even be straight-up statistical methods to do it.
So I think it not the lighting itself that is the biggest problem -- it's having the clean consistent geometries in the first place.
Re: HybridNeRF: Efficient Neural Rendering
#39Absolute noob question that I'm having a hard time understading: In practice, why NeRF instead of Gaussian Splatting? I have very limited exposure to either, but a very cursory search on the subject yields a "it depends on the context" answer. What exact context?
There are two aspects in the difference between NeRF and Gaussian Splatting :
- The first aspect concern how they solve the light rendering equation :
NeRF has more potential for rendering physical quality but is slower.
NeRF use raycasting. Gaussian Splatting project and draw gaussians directly in screen space.
Each have various rendering artefacts. One distinction is in handling light reflections. When you use raycasting, you can bounce your ray on mirror surfaces. Where as gaussian splatting, like alice in wonderland creates a symmetric world on the other side of the mirror (and when the mirror surface is curved, it's hopeless).
Although many NeRF don't implement reflections as a simplification, they can handle them almost natively.
Alternatively, NeRF is a volumetric representation, whereas Gaussian Splatting has surfaces baked in : Gaussian Splats are rendered in order front to back. This mean that when you have two thin objects one behind the other, like the two sides of a book, Gaussian splatting will be able to render the front and hide the back whereas NeRF will merge front and back because volumetric element are transparent. (Though in NeRF with spherical harmonics the Radiance Field direction will allow to cull back from front based on the viewing angle).
- The second aspect of NeRF vs Gaussian Splatting, is the choice of representation :
NeRF usually use a neural network to store the scene in a compressed form. Whereas Gaussian Splatting is more explicit and uncompressed, the scene is represented in a sort of "point cloud" fashion. This mean that if your scene has potential for compression, like repetitive textures or objects, then the NeRF will make use of it and hallucinate what's missing. Whereas gaussian splat will show holes.
Of course like this article is about, you can hybridize them.
Re: HybridNeRF: Efficient Neural Rendering
#40Does anyone else look forward to a game that lets you transform your house or neighbor into a playable level with destructible objects? How far are we from recognizing the “car” and making it drivable, or the “tree” and making it choppable?
An interactive game is much more than just rendering. You need object separation, animation, collision, worldspace, logic, and your requirement of destructibility takes it to a completely different level.
NeRF is not that, it's just a way to represent and render volumetric objects. It's like 10% of what makes a game. Eventually, in theory, it might be possible to make NeRFs or another similar representation animated, interactive, or even entirely drivable by an end-to-end model. But the current state is so far from it that it isn't worth speculating about.
What you want is doable with classic tools already.