Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SMERF: Streamable Memory Efficient Radiance Fields

smerf-3d.github.io

Post reply on HN
Print view View original

121–130 of 152 posts

Re: SMERF: Streamable Memory Efficient Radiance Fields

#121

Post by duckworthd »

nojvek wrote: post #111
duckworthd wrote: post #44

Earlier quoted context omitted.

I can't predict the future, but I imagine soon: all of the tools are there. The reason we didn't develop for VR is actually simpler than you'd think: we just don't have the developer time! At the end of the day, only a handful of people actively wrote code for this project.

Any plans to open source the code?

Yes, I hope so! But it'll take at least a few months of work. We have some tight dependencies to not-yet-open-sourced code, and until that's released, any code we put out will be dead on arrival.

In the meantime, feel free to explore the live viewer code!

https://github.com/smerf-3d/smerf-3d.github.io/blob/main/vie...

Re: SMERF: Streamable Memory Efficient Radiance Fields

#122

Post by 9dev »

edrxty wrote: post #85

Is there any relation between this class of rendering techniques and the way the BD scenes in Cyberpunk 2077 were created? The behavior of the volume and the "voxels" seem eerily similar.

I doubt Cyberpunk uses more than a special shader for the BD sequences, but what’s a lot more remarkable to me is how similar the idea is at heart. Maybe we’re actually going to see this (maybe sans the brain-implant to record them, but hey) after all. Amazing technology, that’s for sure.

Re: SMERF: Streamable Memory Efficient Radiance Fields

#124

Post by vanderZwan »

promiseofbeans wrote: post #98
duckworthd wrote: post #93

Earlier quoted context omitted.

Interesting effect. It does look very voxel-y. I'm not a video game developer at heart, so I can only guess how it was implemented. I doubt NeRF models were involved, but I wouldn't be surprised if some sort of voxel discretization was.

It seems like it might even just be some kind of shader

If you think about how they created this from the POV of the game creation pipeline, then that probably is the way. If this is done by creating a shader on top of "plain old" 3D assets, then aside from the programmers/artists involved with creating that shader everyone else can go about their business with minimal retraining. There probably was a lot of content to create, to that optimization likely took priority over other methods of implementing this effect.

Re: SMERF: Streamable Memory Efficient Radiance Fields

#125

Post by vanderZwan »

duckworthd wrote: post #120
slalomskiing wrote: post #113

I wonder since this runs at real time framerate if it would be possible for someone to composite a regular rasterized frame on top of something like this (with correct depth testing) to make a game For example a 3rd person game where the character you control and the NPCs/enemies is raster but the environment is all radiance fields

This should absolutely be possible! The hard part is making it look natural: NeRF models (including SMERF) have no explicit materials or lighting. That means that any character inserted into the game will look out of place.

Why bother to make it look natural when you can have a really awkward greenscreen-like effect for nostalgic and "artistic" purposes?

Re: SMERF: Streamable Memory Efficient Radiance Fields

#126

Post by strofocles »

duckworthd wrote: post #77
xnx wrote: post #73

Does the an open source toolchain exist for capturing, processing, and hosting navigable 3D walkthroughs like this (e.g. something like an open-source Matterport)?

Not yet, as far as I'm aware. The current flow involves a DSLR for capture, COLMAP for camera parameter estimation, one codebase for training a teacher model, our codebase for training SMERF, and our web viewer for rendering models. Sounds like an opportunity!

Is there a significant advantage for capturing using DSLRs vs using the phone camera of a decent phone?

Re: SMERF: Streamable Memory Efficient Radiance Fields

#127

Post by pavlov »

throwaway17_17 wrote: post #103
pavlov wrote: post #18

Earlier quoted context omitted.

Wow, thanks for the tip. Fridge reflection world is so cool. Feels like something David Lynch might dream up. A girl is eating her morning cereal. Suddenly she looks apprehensively at the fridge. Camera dollies towards the appliance and seamlessly penetrates the reflective surface, revealing a deep hidden space that exactly matches the reflection. At the dark end of the tunnel, something stirs... A wildly grinning ma…

Would you be offended if I animated that scene? It is really well described?

Please feel free!

Re: SMERF: Streamable Memory Efficient Radiance Fields

#128

Post by rpastuszak »

barrkel wrote: post #13

The mirror on the wall of the bathroom in the Berlin location looks through to the kitchen in the next room. I guess the depth gauging algorithm uses parallax, and mirrors confuse it, seeming like windows. The kitchen has a blob of blurriness as the rear of the mirror intrudes into kitchen, but you can see through the blurriness to either room. The effect is a bit spooky. I felt like a ghost going through walls.

Try noclipping through the TV in the Berlin living room. It gets pleasantly creepy.

Re: SMERF: Streamable Memory Efficient Radiance Fields

#130

Post by alkonaut »

nightpool wrote: post #14
barrkel wrote: post #13

The mirror on the wall of the bathroom in the Berlin location looks through to the kitchen in the next room. I guess the depth gauging algorithm uses parallax, and mirrors confuse it, seeming like windows. The kitchen has a blob of blurriness as the rear of the mirror intrudes into kitchen, but you can see through the blurriness to either room. The effect is a bit spooky. I felt like a ghost going through walls.

The refigerator in the NYC scene has a very slick specular lighting effect based on the angle you're viewing it from, and if you go "into" the fridge you can see it's actually generating a whole 3d scene with blurry grey and white colors that turn out to precisely mimic the effects of the light from the windows bouncing off the metal, and you can look "out" from the fridge into the rest of the room. Same as the full-…

What does the reconstructed space look like when there are opposing mirrors? It’ll just be a long corridor of ever more blurry rooms?
Post reply on HN