Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SMERF: Streamable Memory Efficient Radiance Fields

smerf-3d.github.io

Post reply on HN
Print view View original

91–100 of 152 posts

Re: SMERF: Streamable Memory Efficient Radiance Fields

#91

Post by duckworthd »

duckworthd wrote: post #43
VikingCoder wrote: post #16

Wow. Some questions: Take for instance the fulllivingroom demo. (I prefer fps mode.) 1) How many images are input? 2) How long does it take to compute these models? 3) How long does it take to prepare these models for this browser, with all levels, etc? 4) Have you tried this in VR yet?

Glad you liked our work! 1) Around 100-150 if memory serves. This scene is part of the mip-NeRF 360 benchmark, which you can download from the corresponding project website: https://jonbarron.info/mipnerf360/ 2) Between 12 and 48 hours, depending on the scene. We train on 8x V100s or 16x A100s. 3) The time for preparing assets is included in 2). I don't have a breakdown for you, but it's something like 50/50. 4) Nope…

Update: Code for the web viewer is here,

https://github.com/smerf-3d/smerf-3d.github.io/blob/main/vie...

Re: SMERF: Streamable Memory Efficient Radiance Fields

#92

Post by duckworthd »

dougmwne wrote: post #56
duckworthd wrote: post #43

Earlier quoted context omitted.

Glad you liked our work! 1) Around 100-150 if memory serves. This scene is part of the mip-NeRF 360 benchmark, which you can download from the corresponding project website: https://jonbarron.info/mipnerf360/ 2) Between 12 and 48 hours, depending on the scene. We train on 8x V100s or 16x A100s. 3) The time for preparing assets is included in 2). I don't have a breakdown for you, but it's something like 50/50. 4) Nope…

Do you need position data to go along with the photos or just the photos? For VR, there’s going to be some very weird depth data from those reflections, but maybe they would not be so bad when you are in headset.

> Do you need position data to go along with the photos or just the photos?

Short answer: Yes.

Long answer: Yes, but it can typically be derived from images. Structure-from-motion methods are typically used to derive lens and position information for each photo in the training set. These are then used by Zip-NeRF (our teacher) and SMERF (our model) to train a model.

Re: SMERF: Streamable Memory Efficient Radiance Fields

#93

Post by duckworthd »

edrxty wrote: post #89
duckworthd wrote: post #88

Earlier quoted context omitted.

I can't say. I'm not familiar with BD in Cyberpunk.

https://youtu.be/KXXGS3MGCro?t=118 It's a sort of replayable cutscene that happens a couple times in the game where you can wander through it. The noteworthy bit is it's rendered out of voxels that look very similar to the demos but at a much lower resolution and if you push the frustrum into any objects, you get the same kind of effect where the surface breaks into blocks.

Interesting effect. It does look very voxel-y. I'm not a video game developer at heart, so I can only guess how it was implemented. I doubt NeRF models were involved, but I wouldn't be surprised if some sort of voxel discretization was.

Re: SMERF: Streamable Memory Efficient Radiance Fields

#94

Post by duckworthd »

yarg wrote: post #42

What I'm seeing from all of these things is very accurate single navigable 3D images. What I haven't seen anything of is feature and object detection, blocking and extraction. Hopefully a more efficient and streamable codec necessitates the sort of structure that lends itself more easily to analysis.

3D understanding as a field is very much in its infancy. Good work is being done in this area, but we've got a long ways to go yet. SMERF is all about "view synthesis" -- rendering realistic images -- with no attempt at semantic understanding or segmentation.

Re: SMERF: Streamable Memory Efficient Radiance Fields

#95

Post by duckworthd »

smusamashah wrote: post #71

This is very impressive but given its by Google, will some code ever be released?

I hope to release the code in the new year, but we have some big dependencies that need to be released worse. In the meantime, you can already begin hacking on the live viewer, https://github.com/smerf-3d/smerf-3d.github.io/blob/main/vie...

Re: SMERF: Streamable Memory Efficient Radiance Fields

#96

Post by promiseofbeans »

duckworthd wrote: post #40
promiseofbeans wrote: post #15

It runs impressively well on my 2yo s21fe. It was super impressive how it streamed in more images as I explored the space. The tv reflections in the Berlin demo were super impressive. My one note is that it look a really long time to load all the images - the scene wouldn't render until all ~40 initial images loaded. Would it be possible to start partially rendering as the images arrive, or do you need to wait for al…

Pardon our dust: "images" is a bad name for what's being loaded. Past versions of this approach (MERF) stored feature vectors in PNG images. We replace them with binary arrays. Unfortunately, all such arrays need to be loaded before the first frame can be rendered. You do however point out one weakness of SMERF: large payload sizes. If we can figure out how to compress them by 10x, it'll be a very different experienc…

Or even just breaking them down into smaller chunks (prioritise loading the ones closer to where the user is looking) could help

Re: SMERF: Streamable Memory Efficient Radiance Fields

#97

Post by duckworthd »

deelowe wrote: post #69
modeless wrote: post #37

How long until you can stitch Street View into a seamless streaming NeRF of every street in the world? I hope that's the goal you're working towards!

I read another article talking about what waymo was working on and this looks oddly similar... My understanding is that the goal is to use this to reconstruct 3d models of street view images in real time.

Block-NeRF is a predecessor work that helped inspire SMERF, in fact!

https://waymo.com/research/block-nerf/

Re: SMERF: Streamable Memory Efficient Radiance Fields

#98

Post by promiseofbeans »

duckworthd wrote: post #93
edrxty wrote: post #89

Earlier quoted context omitted.

https://youtu.be/KXXGS3MGCro?t=118 It's a sort of replayable cutscene that happens a couple times in the game where you can wander through it. The noteworthy bit is it's rendered out of voxels that look very similar to the demos but at a much lower resolution and if you push the frustrum into any objects, you get the same kind of effect where the surface breaks into blocks.

Interesting effect. It does look very voxel-y. I'm not a video game developer at heart, so I can only guess how it was implemented. I doubt NeRF models were involved, but I wouldn't be surprised if some sort of voxel discretization was.

It seems like it might even just be some kind of shader

Re: SMERF: Streamable Memory Efficient Radiance Fields

#99

Post by duckworthd »

yieldcrv wrote: post #66

I had read about a competing technology that was suggesting NeRF's were a dead end but perhaps that was biased?

You're probably thinking of 3D Gaussian Splatting (3DGS), another fantastic approach to real-time novel view synthesis. There's tons of fantastic work being built on 3DGS right now, and the dust has yet to settle with respect to which method is "better". Right now, I can say that SMERF has slightly higher quality on than 3DGS on small scenes and visibly higher quality on big scenes and runs on a wider variety of devices, but takes much longer than 3DGS to train.

https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/

Re: SMERF: Streamable Memory Efficient Radiance Fields

#100

Post by westurner »

"Researchers create open-source platform for Neural Radiance Field development" (2023) https://news.ycombinator.com/item?id=36966076

NeRF Studio > Included Methods, Third-party Methods: https://docs.nerf.studio/#supported-methods

Neural Radiance Field: https://en.wikipedia.org/wiki/Neural_radiance_field

Post reply on HN