Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SMERF: Streamable Memory Efficient Radiance Fields

smerf-3d.github.io

Post reply on HN
Print view View original

111–120 of 152 posts

Re: SMERF: Streamable Memory Efficient Radiance Fields

#111

Post by nojvek »

duckworthd wrote: post #44
catskul2 wrote: post #30

When might we see this in consumer VR? I'm surprised we don't already but I was suspecting it was a computation constraint. Does this relieve the computation constraint enough to run on Quest 2/3? Is there something else that would prevent binocular use?

I can't predict the future, but I imagine soon: all of the tools are there. The reason we didn't develop for VR is actually simpler than you'd think: we just don't have the developer time! At the end of the day, only a handful of people actively wrote code for this project.

Any plans to open source the code?

Re: SMERF: Streamable Memory Efficient Radiance Fields

#113

Post by slalomskiing »

I wonder since this runs at real time framerate if it would be possible for someone to composite a regular rasterized frame on top of something like this (with correct depth testing) to make a game

For example a 3rd person game where the character you control and the NPCs/enemies is raster but the environment is all radiance fields

Re: SMERF: Streamable Memory Efficient Radiance Fields

#114

Post by jasonwatkinspdx »

slalomskiing wrote: post #113

I wonder since this runs at real time framerate if it would be possible for someone to composite a regular rasterized frame on top of something like this (with correct depth testing) to make a game For example a 3rd person game where the character you control and the NPCs/enemies is raster but the environment is all radiance fields

If you go to one of the demos the space bar will cycle through some debug modes. One shows a surface reconstruction. It comes from the usual structure from motion techniques I presume, so it's coarse and noisy, but I think the fundamental idea is viable.

Re: SMERF: Streamable Memory Efficient Radiance Fields

#115

Post by duckworthd »

nojvek wrote: post #110

Holy mother of god. Wow! Either matterport takes and runs with this or this is a startup waiting to disrupt Realestate. I can’t believe how smooth this ran on my smartphone. Feedback: if there was a mode to use the phone compass and gyro for navigation, it’d feel natural. Felt weird to navigate with fingers and figure how to move in xyz dimension. As others have said, VR mode would be epic.

Thanks for the feedback!

I agree, we could do better with the movement UX. A challenge for another day.

Re: SMERF: Streamable Memory Efficient Radiance Fields

#116

Post by duckworthd »

promiseofbeans wrote: post #96
duckworthd wrote: post #40

Earlier quoted context omitted.

Pardon our dust: "images" is a bad name for what's being loaded. Past versions of this approach (MERF) stored feature vectors in PNG images. We replace them with binary arrays. Unfortunately, all such arrays need to be loaded before the first frame can be rendered. You do however point out one weakness of SMERF: large payload sizes. If we can figure out how to compress them by 10x, it'll be a very different experienc…

Or even just breaking them down into smaller chunks (prioritise loading the ones closer to where the user is looking) could help

The viewer biases towards assets closer to user's camera (otherwise you'd have to load the whole scene!). We tried training SMERF with a larger number of smaller submodels, but at some point, it becomes too onerous to train and quality begins to suffer.

Re: SMERF: Streamable Memory Efficient Radiance Fields

#117

Post by duckworthd »

nh2 wrote: post #102
duckworthd wrote: post #91

Earlier quoted context omitted.

Update: Code for the web viewer is here, https://github.com/smerf-3d/smerf-3d.github.io/blob/main/vie...

What is the license? The repo doesn't say.

Oops, I need to update the license files.

Our code is released under the Apache 2.0 license, as in this repo: https://github.com/google-research/google-research/blob/mast...

Re: SMERF: Streamable Memory Efficient Radiance Fields

#118

Post by duckworthd »

zyang wrote: post #105

Why is there a 300m^2 footprint limit if the sub-models are dynamically loaded. Is this constrained by training, rasterizing, or both?

In terms of the live viewer, there's actually no limit on footprint size. 300 m^2 is simply the biggest indoor capture we had!

Re: SMERF: Streamable Memory Efficient Radiance Fields

#119

Post by duckworthd »

digdugdirk wrote: post #112

Can you recommend a good entry point into the theory/math behind these? This is one of those true "wtf, we can do this now?" moments, I'm super curious about how these are generated/created.

Oof, there's a lot of machinery here. It depends a lot on your academic background.

I'd recommend starting with a tutorial on neural radiance fields, aka NeRF, (https://sites.google.com/berkeley.edu/nerf-tutorial/home) and an applied overview of Deep Learning with tools like PyTorch or JAX. This line of work is still "cutting edge" research, so a lot of knowledge hasn't been rolled up into textbook or article form yet.

Re: SMERF: Streamable Memory Efficient Radiance Fields

#120

Post by duckworthd »

slalomskiing wrote: post #113

I wonder since this runs at real time framerate if it would be possible for someone to composite a regular rasterized frame on top of something like this (with correct depth testing) to make a game For example a 3rd person game where the character you control and the NPCs/enemies is raster but the environment is all radiance fields

This should absolutely be possible! The hard part is making it look natural: NeRF models (including SMERF) have no explicit materials or lighting. That means that any character inserted into the game will look out of place.
Post reply on HN