NeRF: Representing scenes as neural radiance fields for view synthesis • Hacker News Board

Re: NeRF: Representing scenes as neural radiance fields for view synthesis

#11

Post by scribu » Fri, Mar 20, 2020, 5:07 PM UTC

ssivark wrote: post #9

jayd16 wrote: post #7

Very cool. Reminds me of when I played with Google's Seurat. The paper says its 5MB, 12 hours to train the NN and then 30 seconds to render novel views of the scene on an nVidia V100. Sadly not something you can use in real time but still very cool. Edit:12 hours and 5MB NN not 5 Minutes

Huh, what? It needs almost a million views, and takes 1-2 days to train on a GPU. I’m not sure where the “5 minutes” number comes from. EDIT: I was referring to the last paragraph of section 5.3 (Implementation details), but maybe I’m misunderstanding how they use rays / sampled coordinates. Very impressive visual quality. But it seems like they need a LOT of data and computation for each scene. So, its still plausib…

> It needs almost a million views

Not sure what you mean by "views". The comparisons in the paper use at most 100 input images per scene.

Re: NeRF: Representing scenes as neural radiance fields for view synthesis

#12

Post by visarga » Fri, Mar 20, 2020, 5:09 PM UTC

wokwokwok wrote: post #4

quadrature wrote: post #3

Earlier quoted context omitted.

its a very similar concept to photogrammetry which is recovering a 3d representation of an object given pictures taken from different angles. In this work they take pictures of a scene from different angles and are able to train a neural network to render the scene from new angles that aren't in any source pictures. The neural network takes in a location (x,y,z), a viewing direction and spits out the RGB of the rende…

Significantly, the input is a sparse dataset. ie. Few source images vs. traditional photogrammetry. ...but basically yes, tldr; photogrammetry using neural networks; this one is better than other recent attempts at the same thing, but takes a really long time (2 days for this vs 10 minutes for a voxel based approach in one of their comparisons). Why bother? mmm... theres some kind speculation you might be able to rep…

> Why bother?

There might be 10x speedups to be gained with a tweaked model.

Re: NeRF: Representing scenes as neural radiance fields for view synthesis

#13

Post by kuprel » Fri, Mar 20, 2020, 5:19 PM UTC

This would be great for instant replays

Re: NeRF: Representing scenes as neural radiance fields for view synthesis

#14

Post by jayd16 » Fri, Mar 20, 2020, 5:24 PM UTC

kuprel wrote: post #13

This would be great for instant replays

Intel already does this with their "True View" setup. They also had a tech demo CES where they synthesized camera positions for movie sets. https://www.youtube.com/watch?v=9qd276AJg-o

Re: NeRF: Representing scenes as neural radiance fields for view synthesis

#15

Post by jayd16 » Fri, Mar 20, 2020, 5:28 PM UTC

ssivark wrote: post #9

jayd16 wrote: post #7

Very cool. Reminds me of when I played with Google's Seurat. The paper says its 5MB, 12 hours to train the NN and then 30 seconds to render novel views of the scene on an nVidia V100. Sadly not something you can use in real time but still very cool. Edit:12 hours and 5MB NN not 5 Minutes

Huh, what? It needs almost a million views, and takes 1-2 days to train on a GPU. I’m not sure where the “5 minutes” number comes from. EDIT: I was referring to the last paragraph of section 5.3 (Implementation details), but maybe I’m misunderstanding how they use rays / sampled coordinates. Very impressive visual quality. But it seems like they need a LOT of data and computation for each scene. So, its still plausib…

Excuse me I meant 5MB. It takes 12 hours to train.

>All compared single scene methods take at least 12 hours to train per scene

But it seems to only need sparse images.

>Here, we visualize the set of 100 input views of the synthetic Drums scene randomly captured on a surrounding hemisphere, and we show two novel views rendered from our optimized NeRF representation

Re: NeRF: Representing scenes as neural radiance fields for view synthesis

#16

Post by byt143 » Fri, Mar 20, 2020, 5:55 PM UTC

If you're only looking for one novel view, can it use less views that are close to the novel one?

Re: NeRF: Representing scenes as neural radiance fields for view synthesis

#17

Post by rebuilder » Fri, Mar 20, 2020, 6:05 PM UTC

wokwokwok wrote: post #4

quadrature wrote: post #3

Earlier quoted context omitted.

its a very similar concept to photogrammetry which is recovering a 3d representation of an object given pictures taken from different angles. In this work they take pictures of a scene from different angles and are able to train a neural network to render the scene from new angles that aren't in any source pictures. The neural network takes in a location (x,y,z), a viewing direction and spits out the RGB of the rende…

Significantly, the input is a sparse dataset. ie. Few source images vs. traditional photogrammetry. ...but basically yes, tldr; photogrammetry using neural networks; this one is better than other recent attempts at the same thing, but takes a really long time (2 days for this vs 10 minutes for a voxel based approach in one of their comparisons). Why bother? mmm... theres some kind speculation you might be able to rep…

A number of things this seems to do well would be pretty much impossible with standard photogrammetry : trees with leaves, fine details like rigging on a ship, reflective surfaces, even refraction (!)

Of course the output is a new view, not a shaded mesh, but given it appears to generate depth data, I think you should be able to generate a point cloud and mesh it. Getting the materials from the output light even be possible, I'm not very up to date on the state of material capture nowadays.

Re: NeRF: Representing scenes as neural radiance fields for view synthesis

#18

Post by ssivark » Fri, Mar 20, 2020, 6:28 PM UTC

Does anyone know how they do the “virtual object insertion” demonstrated in the paper summary video? Can that be somehow done on the network itself, or is that a diagnostic for scene accuracy by performing SFM on network output?

Re: NeRF: Representing scenes as neural radiance fields for view synthesis

#19

Post by notfed » Fri, Mar 20, 2020, 7:16 PM UTC

mooneater wrote: post #6

blackhaz wrote: post #2

Could someone ELI5, please?

If you give it a bunch of photos of a scene from different angles, this machine learning method lets you see angles that did not exist in the original set. Better results than other methods so far.

Fist bump for actually answering as ELI5 (unlike the other responses).

Re: NeRF: Representing scenes as neural radiance fields for view synthesis

#20

Post by theresistor » Fri, Mar 20, 2020, 8:23 PM UTC

ssivark wrote: post #18

Does anyone know how they do the “virtual object insertion” demonstrated in the paper summary video? Can that be somehow done on the network itself, or is that a diagnostic for scene accuracy by performing SFM on network output?

I'm pretty sure they're rendering a depth channel and compositing it in.