Nerfstudio: A collaboration friendly studio for NeRFs • Hacker News Board

Re: Nerfstudio: A collaboration friendly studio for NeRFs

#21

Post by amelius » Wed, Oct 05, 2022, 11:39 PM UTC

gorkish wrote: post #6

I am extremely excited for this technology to mature to a point where realtime novel view synthesis becomes possible. The idea of being able to take a few photographs and be able to recreate an appreciably decent 6DOF viewing experience is a very powerful enabling technology. Nerfstudio looks like something that is finally accessible enough that I will be able to start experimenting without having to spend enormous a…

> The idea of being able to take a few photographs and be able to recreate an appreciably decent 6DOF viewing experience is a very powerful enabling technology.

My understanding is that you can't just take a few photographs, but you have to label them with their position and orientation, which is often not trivial.

But perhaps this difficulty has been solved already (?)

Re: Nerfstudio: A collaboration friendly studio for NeRFs

#22

Post by krasin » Wed, Oct 05, 2022, 11:43 PM UTC

I tried it today. Nerfstudio is amazing. I also like the pragmatic approach to its UI: launch a command-line / text UI program, publish a localhost URL, access a rich WebUI in the browser.

NeRF rendering latency is extremely low and frames are delivered over WebRTC. Which means that there's no fundamental problem to stream WebUI over the Internet.

Nerfstudio already supports all leading NeRF flavours and also includes their own. It instantly makes a staple for all future NeRF research.

Re: Nerfstudio: A collaboration friendly studio for NeRFs

#23

Post by krasin » Thu, Oct 06, 2022, 12:08 AM UTC

amelius wrote: post #21

gorkish wrote: post #6

I am extremely excited for this technology to mature to a point where realtime novel view synthesis becomes possible. The idea of being able to take a few photographs and be able to recreate an appreciably decent 6DOF viewing experience is a very powerful enabling technology. Nerfstudio looks like something that is finally accessible enough that I will be able to start experimenting without having to spend enormous a…

> The idea of being able to take a few photographs and be able to recreate an appreciably decent 6DOF viewing experience is a very powerful enabling technology. My understanding is that you can't just take a few photographs, but you have to label them with their position and orientation, which is often not trivial. But perhaps this difficulty has been solved already (?)

Usually, you can just run COLMAP ([1]) and it will find the poses and camera intrinsics. This is indeed what Nerfstudio recommends doing in the docs ([2]).

1. https://colmap.github.io/

2. https://docs.nerf.studio/en/latest/quickstart/custom_dataset...

Re: Nerfstudio: A collaboration friendly studio for NeRFs

#24

Post by yalue » Thu, Oct 06, 2022, 12:21 AM UTC

mshockwave wrote: post #17

bashinator wrote: post #10

It would be helpful if that site didn't just assume all visitors already know what a NeRF is.

exactly, can somebody ELI5 what is NeRF in this context?

At least in the original paper [1], the key idea is that instead of training a neural network to treat "images" as an array of pixels, you train a network to map a camera location and direction (in 3D space) to a distance and a color.

For example, if you tell a NeRF that you have a camera at location (x, y, z) and pointing in direction (g, h, j), then the network will output the distance at which the ray emitted from the camera is expected to "hit" something and the RGB color of whatever is hit.

Doing things in this way enables rendering images at arbitrary resolutions (though rendering can be slow), and is naturally conducive to producing rotated views of objects or exploring 3D space. Also, at least theoretically, it should allow for more "compact" network architectures, as it does not need to output, say, a 512x512x3 image.

[1] https://arxiv.org/pdf/2003.08934.pdf

Re: Nerfstudio: A collaboration friendly studio for NeRFs

#25

Post by moron4hire » Thu, Oct 06, 2022, 12:48 AM UTC

gorkish wrote: post #6

I am extremely excited for this technology to mature to a point where realtime novel view synthesis becomes possible. The idea of being able to take a few photographs and be able to recreate an appreciably decent 6DOF viewing experience is a very powerful enabling technology. Nerfstudio looks like something that is finally accessible enough that I will be able to start experimenting without having to spend enormous a…

Real time novel view synthesis is already possible. In fact, I recently saw someone demo a version that ran in WebGL2 shaders on smartphones, so it's not terribly taxing.

https://mobile-nerf.github.io/

https://mobilenerf-webxr.fern.solutions/

Re: Nerfstudio: A collaboration friendly studio for NeRFs

#26

Post by tunesmith » Thu, Oct 06, 2022, 1:34 AM UTC

karxxm wrote: post #18

mshockwave wrote: post #17

Earlier quoted context omitted.

exactly, can somebody ELI5 what is NeRF in this context?

A NeRF is a deep neural network architecture, that takes a set of photos from multiple camera positions of a given scene as an input and outputs a volume representation of this scene. Volumes allow for many common volume rendering interactions like repositioning the camera from any angle or even lighting effects.

Like the Blade Runner camera!

Re: Nerfstudio: A collaboration friendly studio for NeRFs

#27

Post by andybak » Thu, Oct 06, 2022, 12:15 PM UTC

bohadi wrote: post #15

NeRF applied to Maps soon https://twitter.com/jeffdean/status/1524612819160735745?lang... another paper toward Earth sized multi scale models (5 minutes) https://www.youtube.com/watch?v=Pn9ia2fsiwY

So a bunch of news sites have reported this feature has already rolled out. For example: https://www.engadget.com/google-maps-aerial-view-landmarks-0...

And lo and behold I see something that looks like a 3d rendered view at the top of the entry for "Tower of London".

Except - it's just a video. A rendered video of a 360 view. Tried on Android and iPad. There's no freedom of movement, it plays a single look and even says "report this video" when you click the 3 dots.

Are the news reports wrong - or do the writers not know a video when they see one?

Re: Nerfstudio: A collaboration friendly studio for NeRFs

#28

Post by gorkish » Wed, Oct 12, 2022, 8:23 PM UTC

moron4hire wrote: post #25

gorkish wrote: post #6

I am extremely excited for this technology to mature to a point where realtime novel view synthesis becomes possible. The idea of being able to take a few photographs and be able to recreate an appreciably decent 6DOF viewing experience is a very powerful enabling technology. Nerfstudio looks like something that is finally accessible enough that I will be able to start experimenting without having to spend enormous a…

Real time novel view synthesis is already possible. In fact, I recently saw someone demo a version that ran in WebGL2 shaders on smartphones, so it's not terribly taxing. https://mobile-nerf.github.io/ https://mobilenerf-webxr.fern.solutions/

Thanks very much for the reference. I'm curious about this but unfortunately not enough to keep up with the nooks and crannies where it's being worked on!