Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SMERF: Streamable Memory Efficient Radiance Fields

smerf-3d.github.io

Post reply on HN
Print view View original

141–150 of 152 posts

Re: SMERF: Streamable Memory Efficient Radiance Fields

#141

Post by pjc50 »

nightpool wrote: post #14
barrkel wrote: post #13

The mirror on the wall of the bathroom in the Berlin location looks through to the kitchen in the next room. I guess the depth gauging algorithm uses parallax, and mirrors confuse it, seeming like windows. The kitchen has a blob of blurriness as the rear of the mirror intrudes into kitchen, but you can see through the blurriness to either room. The effect is a bit spooky. I felt like a ghost going through walls.

The refigerator in the NYC scene has a very slick specular lighting effect based on the angle you're viewing it from, and if you go "into" the fridge you can see it's actually generating a whole 3d scene with blurry grey and white colors that turn out to precisely mimic the effects of the light from the windows bouncing off the metal, and you can look "out" from the fridge into the rest of the room. Same as the full-…

Funnily enough, this is how reflections are usually emulated in game engines that do not support raytracing: another copy of the world behind the mirror. Also used in films in a few places (e.g. Terminator)

Re: SMERF: Streamable Memory Efficient Radiance Fields

#143

Post by hobofan »

tobr wrote: post #138
nojvek wrote: post #110

Holy mother of god. Wow! Either matterport takes and runs with this or this is a startup waiting to disrupt Realestate. I can’t believe how smooth this ran on my smartphone. Feedback: if there was a mode to use the phone compass and gyro for navigation, it’d feel natural. Felt weird to navigate with fingers and figure how to move in xyz dimension. As others have said, VR mode would be epic.

Is this really something the real estate market wants though? The point of using styled and meticulously chosen images is to entice people to visit the property in person. I think it’s hard to fall for a home because you saw it through virtual reality.

> I think it’s hard to fall for a home because you saw it through virtual reality.

I think if you take this 1-2 steps further and combine this with halucinating already owned furniture, or furniture that matches the prospective buyers taste into the property, this will make it a lot easier to fall for a home.

Re: SMERF: Streamable Memory Efficient Radiance Fields

#144

Post by nightpool »

pjc50 wrote: post #141
nightpool wrote: post #14

Earlier quoted context omitted.

The refigerator in the NYC scene has a very slick specular lighting effect based on the angle you're viewing it from, and if you go "into" the fridge you can see it's actually generating a whole 3d scene with blurry grey and white colors that turn out to precisely mimic the effects of the light from the windows bouncing off the metal, and you can look "out" from the fridge into the rest of the room. Same as the full-…

Funnily enough, this is how reflections are usually emulated in game engines that do not support raytracing: another copy of the world behind the mirror. Also used in films in a few places (e.g. Terminator )

Please look at the refrigerator I mentioned—it's definitely not the classic "mirror world" reflection that you'd normally see in video games. I'm talking about the specular / metallic highlights on the fridge being simulated entirely with depth features.

Re: SMERF: Streamable Memory Efficient Radiance Fields

#146

Post by ilaksh »

duckworthd wrote: post #83
ilaksh wrote: post #63

Earlier quoted context omitted.

Will the code be released, or an API endpoint? Otherwise it will be impossible for us to use it for anything.. since it's Google I assume it will just end up in a black hole like most of the research.. or five years later some AI researchers leave and finally create a startup.

I hope to release code in the new year, but it'll take a while. The codebase is heavily wired into other not-yet-open-sourced libraries, and it'll take a while to disentangle them.

That sounds terrific! I really appreciate your effort. It's amazing work and so great of you to share it.

Re: SMERF: Streamable Memory Efficient Radiance Fields

#147

Post by goldenshale »

yarg wrote: post #42

What I'm seeing from all of these things is very accurate single navigable 3D images. What I haven't seen anything of is feature and object detection, blocking and extraction. Hopefully a more efficient and streamable codec necessitates the sort of structure that lends itself more easily to analysis.

Checkout the LERF work from the NerfStudio team at UC Berkeley. SMERF is addressing a different problem, but there are definitely ways to incorporate semantics and detection as well.

Re: SMERF: Streamable Memory Efficient Radiance Fields

#148

Post by asgerhb »

duckworthd wrote: post #133
asgerhb wrote: post #129

Wow! What am I even looking at here? Polygons, voxels, or something else entirely? How were the benchmarks recorded?

You're looking at something called a "neural radiance field" backed by a sparse, low resolution voxel grid and a dense high resolution triplane grid. That's a bit of a word soup, but you can think of it like a glowing fog rendered with ray marching. The benchmark details are a bit complicated. Check out the technical paper's experiment section for the nitty gritty details.

Hahah I'm in formal methods, not graphics, so I'm picturing a literal green glowing fog. But what I'm gathering is polygons partitioned into cube chunks. This is of course not including the particulars of this impressive contribution, but I fear that's beyond my ken.

Re: SMERF: Streamable Memory Efficient Radiance Fields

#149

Post by CuriousCosmic »

ibrarmalik wrote: post #60
CuriousCosmic wrote: post #12

Is there a relatively easy way to apply these kinds of techniques (either NeRFs or gaussian splats) to larger environments even if it's lower precision? Like say small towns/a few blocks worth of env.

You’re under the right paper for doing this. Instead of one big model, they have several smaller ones for regions in the scene. This way rendering is fast for large scenes. This is similar to Block-NeRF [0], in their project page they show some videos of what you’re asking. As for an easy way of doing this, nothing out-of-the-box. You can keep an eye on nerfstudio [1], and if you feel brave you could implement this p…

Oooh fun. I'm glad it seems possible nowadays. I might take a swing at putting together an out of the box tool at some point if nobody beats me to it first.
Post reply on HN