Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SMERF: Streamable Memory Efficient Radiance Fields

smerf-3d.github.io

Post reply on HN
Print view View original

131–140 of 152 posts

Re: SMERF: Streamable Memory Efficient Radiance Fields

#131

Post by yorwba »

yarg wrote: post #42

What I'm seeing from all of these things is very accurate single navigable 3D images. What I haven't seen anything of is feature and object detection, blocking and extraction. Hopefully a more efficient and streamable codec necessitates the sort of structure that lends itself more easily to analysis.

You mean something like this? https://jumpat.github.io/SA3D/

Found by putting "nerf sam segment 3d" into DuckDuckGo.

Re: SMERF: Streamable Memory Efficient Radiance Fields

#132

Post by nojvek »

duckworthd wrote: post #115
nojvek wrote: post #110

Holy mother of god. Wow! Either matterport takes and runs with this or this is a startup waiting to disrupt Realestate. I can’t believe how smooth this ran on my smartphone. Feedback: if there was a mode to use the phone compass and gyro for navigation, it’d feel natural. Felt weird to navigate with fingers and figure how to move in xyz dimension. As others have said, VR mode would be epic.

Thanks for the feedback! I agree, we could do better with the movement UX. A challenge for another day.

Since the viewer is on GitHub, I’ll take it for a spin.

Are you accepting pull requests?

Re: SMERF: Streamable Memory Efficient Radiance Fields

#133

Post by duckworthd »

asgerhb wrote: post #129

Wow! What am I even looking at here? Polygons, voxels, or something else entirely? How were the benchmarks recorded?

You're looking at something called a "neural radiance field" backed by a sparse, low resolution voxel grid and a dense high resolution triplane grid. That's a bit of a word soup, but you can think of it like a glowing fog rendered with ray marching.

The benchmark details are a bit complicated. Check out the technical paper's experiment section for the nitty gritty details.

Re: SMERF: Streamable Memory Efficient Radiance Fields

#134

Post by duckworthd »

strofocles wrote: post #126
duckworthd wrote: post #77

Earlier quoted context omitted.

Not yet, as far as I'm aware. The current flow involves a DSLR for capture, COLMAP for camera parameter estimation, one codebase for training a teacher model, our codebase for training SMERF, and our web viewer for rendering models. Sounds like an opportunity!

Is there a significant advantage for capturing using DSLRs vs using the phone camera of a decent phone?

The big difference is access to fisheye lenses a burst mode that can be run for minutes at a time, and the ability to minimize the amount of camera post processing. In principle, the capture could be done with a smartphone, but the experience of doing so is pretty time consuming right now.

Re: SMERF: Streamable Memory Efficient Radiance Fields

#135

Post by duckworthd »

zyang wrote: post #109
duckworthd wrote: post #78

Earlier quoted context omitted.

All these details and more in our technical paper! In short: SMERF training takes much longer, SMERF rendering is nearly as fast as 3DGS when a CUDA GPU is available, and quality is visibly higher than 3DGS on large scenes and slightly higher on smaller scenes. https://arxiv.org/abs/2312.07541

Is it possible to use zip-nerf to train GS to eliminate the floaters.

Maybe! That's the seed of a completely different research paper :)

Re: SMERF: Streamable Memory Efficient Radiance Fields

#136

Post by duckworthd »

azinman2 wrote: post #123

Will there be any notebooks or other code released to train our own models?

I hope so, but it'll be a good while before we can release anything. We have tight dependencies to other not-yet-OSS libraries, and until they're released, our won't work either.

Re: SMERF: Streamable Memory Efficient Radiance Fields

#138

Post by tobr »

nojvek wrote: post #110

Holy mother of god. Wow! Either matterport takes and runs with this or this is a startup waiting to disrupt Realestate. I can’t believe how smooth this ran on my smartphone. Feedback: if there was a mode to use the phone compass and gyro for navigation, it’d feel natural. Felt weird to navigate with fingers and figure how to move in xyz dimension. As others have said, VR mode would be epic.

Is this really something the real estate market wants though? The point of using styled and meticulously chosen images is to entice people to visit the property in person. I think it’s hard to fall for a home because you saw it through virtual reality.

Re: SMERF: Streamable Memory Efficient Radiance Fields

#140

Post by iandanforth »

tobr wrote: post #138
nojvek wrote: post #110

Holy mother of god. Wow! Either matterport takes and runs with this or this is a startup waiting to disrupt Realestate. I can’t believe how smooth this ran on my smartphone. Feedback: if there was a mode to use the phone compass and gyro for navigation, it’d feel natural. Felt weird to navigate with fingers and figure how to move in xyz dimension. As others have said, VR mode would be epic.

Is this really something the real estate market wants though? The point of using styled and meticulously chosen images is to entice people to visit the property in person. I think it’s hard to fall for a home because you saw it through virtual reality.

It is, yes. If you browse zillow you'll find many homes have 3D views attached. These are often image-sphere captures that you can painfully move through by clicking. While I agree that full res photos can be more appealing, the user experience with SMERF is so much better it might leave end users with a more positive feeling about a property and thus increase the chances of a sale.
Post reply on HN