51–60 of 152 posts
Any plans to do this in VR? I would love to try this.
UPDATE: The code for our web viewer is here: https://github.com/smerf-3d/smerf-3d.github.io/blob/main/vie...
There is a market here for Realtors to upload pictures and produce walk-throughs of homes for sale.
This is __really__ stunning work, huge, huge, deal that I'm seeing this in a web browser on my phone. Congratulations! When I look at the NYC scene in the highest quality on desktop, I'm surprised by how low-quality ex. the stuff on the counter and shelves is. So then I load the lego model, and see that's _very_ detailed, so it doesn't seem inherent to the method. Is it a consequence of input photo quality, or someth…
Thank you :)
> Is it a consequence of input photo quality, or something else?
It's more a consequence of spatial resolution: the bigger the space, the more voxels you need to maintain a fixed resolution (e.g. 1 mm^3). At some point, we have to give up spatial resolution to represent larger scenes.
A second limitation is the teacher model we're distilling. Zip-NeRF (https://jonbarron.info/zipnerf/) is good, but it's not _perfect_. SMERF reconstruction quality is upper-bounded by its Zip-NeRF teacher.
I'm following this through two minutes paper and I'm looking forward to using it. My grandpa died 2 years ago and in hindsight I took pictures for using them as in your demo. Awesome thanks:)
Wow. Some questions: Take for instance the fulllivingroom demo. (I prefer fps mode.) 1) How many images are input? 2) How long does it take to compute these models? 3) How long does it take to prepare these models for this browser, with all levels, etc? 4) Have you tried this in VR yet?
Glad you liked our work! 1) Around 100-150 if memory serves. This scene is part of the mip-NeRF 360 benchmark, which you can download from the corresponding project website: https://jonbarron.info/mipnerf360/ 2) Between 12 and 48 hours, depending on the scene. We train on 8x V100s or 16x A100s. 3) The time for preparing assets is included in 2). I don't have a breakdown for you, but it's something like 50/50. 4) Nope…
For VR, there’s going to be some very weird depth data from those reflections, but maybe they would not be so bad when you are in headset.
The mirror on the wall of the bathroom in the Berlin location looks through to the kitchen in the next room. I guess the depth gauging algorithm uses parallax, and mirrors confuse it, seeming like windows. The kitchen has a blob of blurriness as the rear of the mirror intrudes into kitchen, but you can see through the blurriness to either room. The effect is a bit spooky. I felt like a ghost going through walls.
The mirror on the wall of the bathroom in the Berlin location looks through to the kitchen in the next room. I guess the depth gauging algorithm uses parallax, and mirrors confuse it, seeming like windows. The kitchen has a blob of blurriness as the rear of the mirror intrudes into kitchen, but you can see through the blurriness to either room. The effect is a bit spooky. I felt like a ghost going through walls.
How long until you can stitch Street View into a seamless streaming NeRF of every street in the world? I hope that's the goal you're working towards!
;)
Is there a relatively easy way to apply these kinds of techniques (either NeRFs or gaussian splats) to larger environments even if it's lower precision? Like say small towns/a few blocks worth of env.
This is similar to Block-NeRF [0], in their project page they show some videos of what you’re asking.
As for an easy way of doing this, nothing out-of-the-box. You can keep an eye on nerfstudio [1], and if you feel brave you could implement this paper and make a PR!