NeRF: Representing scenes as neural radiance fields for view synthesis
1–10 of 43 posts
Re: NeRF: Representing scenes as neural radiance fields for view synthesis
#2Could someone ELI5, please?
Re: NeRF: Representing scenes as neural radiance fields for view synthesis
#3Could someone ELI5, please?
its a very similar concept to photogrammetry which is recovering a 3d representation of an object given pictures taken from different angles.
In this work they take pictures of a scene from different angles and are able to train a neural network to render the scene from new angles that aren't in any source pictures.
The neural network takes in a location (x,y,z), a viewing direction and spits out the RGB of the rendered image if you were to view the scene at that location and angle.
Using this network and traditional rendering techniques they are able to render the whole scene.
Re: NeRF: Representing scenes as neural radiance fields for view synthesis
#4Could someone ELI5, please?
its a very similar concept to photogrammetry which is recovering a 3d representation of an object given pictures taken from different angles. In this work they take pictures of a scene from different angles and are able to train a neural network to render the scene from new angles that aren't in any source pictures. The neural network takes in a location (x,y,z), a viewing direction and spits out the RGB of the rende…
Significantly, the input is a sparse dataset.
ie. Few source images vs. traditional photogrammetry.
...but basically yes, tldr; photogrammetry using neural networks; this one is better than other recent attempts at the same thing, but takes a really long time (2 days for this vs 10 minutes for a voxel based approach in one of their comparisons).
Why bother?
mmm... theres some kind speculation you might be able to represent a photorealistic scene/ 3d object as a neural model instead of voxels or meshes.
That might be useful for some things. eg. say, a voxel representation of semi transparent fog, or high detail objects like hair are impractically huge, and as a mesh its very difficult to represent.
Re: NeRF: Representing scenes as neural radiance fields for view synthesis
#5Could someone ELI5, please?
They're modeling a scene mathematically as a "radiance field" - a function that takes a view position and direction as inputs and returns the light color that hits that position from the direction it's facing. They use some input images to train a neural network, in order to find an optimal radiance field function which explains the input images. Once they have that function, they can construct images from new angles by evaluating the function over the (position, direction) inputs needed by the pixels in the new image.
Re: NeRF: Representing scenes as neural radiance fields for view synthesis
#6Could someone ELI5, please?
If you give it a bunch of photos of a scene from different angles, this machine learning method lets you see angles that did not exist in the original set.
Better results than other methods so far.
Re: NeRF: Representing scenes as neural radiance fields for view synthesis
#7Very cool. Reminds me of when I played with Google's Seurat.
The paper says its 5MB, 12 hours to train the NN and then 30 seconds to render novel views of the scene on an nVidia V100.
Sadly not something you can use in real time but still very cool.
Edit:12 hours and 5MB NN not 5 Minutes
Re: NeRF: Representing scenes as neural radiance fields for view synthesis
#8[deleted]
Re: NeRF: Representing scenes as neural radiance fields for view synthesis
#9Very cool. Reminds me of when I played with Google's Seurat. The paper says its 5MB, 12 hours to train the NN and then 30 seconds to render novel views of the scene on an nVidia V100. Sadly not something you can use in real time but still very cool. Edit:12 hours and 5MB NN not 5 Minutes
Huh, what? It needs almost a million views, and takes 1-2 days to train on a GPU. I’m not sure where the “5 minutes” number comes from.
EDIT: I was referring to the last paragraph of section 5.3 (Implementation details), but maybe I’m misunderstanding how they use rays / sampled coordinates.
Very impressive visual quality. But it seems like they need a LOT of data and computation for each scene. So, its still plausible that intelligently done photogrammetry will beat this approach in efficiency, but a bunch of important details need to be figured out to make that happen.
Re: NeRF: Representing scenes as neural radiance fields for view synthesis
#10This is bad-ass, partly because it's so elegant.