101–110 of 152 posts
Earlier quoted context omitted.
I read another article talking about what waymo was working on and this looks oddly similar... My understanding is that the goal is to use this to reconstruct 3d models of street view images in real time.
Block-NeRF is a predecessor work that helped inspire SMERF, in fact! https://waymo.com/research/block-nerf/
Earlier quoted context omitted.
Glad you liked our work! 1) Around 100-150 if memory serves. This scene is part of the mip-NeRF 360 benchmark, which you can download from the corresponding project website: https://jonbarron.info/mipnerf360/ 2) Between 12 and 48 hours, depending on the scene. We train on 8x V100s or 16x A100s. 3) The time for preparing assets is included in 2). I don't have a breakdown for you, but it's something like 50/50. 4) Nope…
Update: Code for the web viewer is here, https://github.com/smerf-3d/smerf-3d.github.io/blob/main/vie...
Earlier quoted context omitted.
The refigerator in the NYC scene has a very slick specular lighting effect based on the angle you're viewing it from, and if you go "into" the fridge you can see it's actually generating a whole 3d scene with blurry grey and white colors that turn out to precisely mimic the effects of the light from the windows bouncing off the metal, and you can look "out" from the fridge into the rest of the room. Same as the full-…
Wow, thanks for the tip. Fridge reflection world is so cool. Feels like something David Lynch might dream up. A girl is eating her morning cereal. Suddenly she looks apprehensively at the fridge. Camera dollies towards the appliance and seamlessly penetrates the reflective surface, revealing a deep hidden space that exactly matches the reflection. At the dark end of the tunnel, something stirs... A wildly grinning ma…
What I'm seeing from all of these things is very accurate single navigable 3D images. What I haven't seen anything of is feature and object detection, blocking and extraction. Hopefully a more efficient and streamable codec necessitates the sort of structure that lends itself more easily to analysis.
3D understanding as a field is very much in its infancy. Good work is being done in this area, but we've got a long ways to go yet. SMERF is all about "view synthesis" -- rendering realistic images -- with no attempt at semantic understanding or segmentation.
It is funny how quickly goalposts move! I love to see progress though, and wow, is progress happening fast!
The mirror on the wall of the bathroom in the Berlin location looks through to the kitchen in the next room. I guess the depth gauging algorithm uses parallax, and mirrors confuse it, seeming like windows. The kitchen has a blob of blurriness as the rear of the mirror intrudes into kitchen, but you can see through the blurriness to either room. The effect is a bit spooky. I felt like a ghost going through walls.
The mirror on the wall of the bathroom in the Berlin location looks through to the kitchen in the next room. I guess the depth gauging algorithm uses parallax, and mirrors confuse it, seeming like windows. The kitchen has a blob of blurriness as the rear of the mirror intrudes into kitchen, but you can see through the blurriness to either room. The effect is a bit spooky. I felt like a ghost going through walls.
The refigerator in the NYC scene has a very slick specular lighting effect based on the angle you're viewing it from, and if you go "into" the fridge you can see it's actually generating a whole 3d scene with blurry grey and white colors that turn out to precisely mimic the effects of the light from the windows bouncing off the metal, and you can look "out" from the fridge into the rest of the room. Same as the full-…
The barely-there reflection on the Berlin TV is also a trip to enter, and observe the room from.
Earlier quoted context omitted.
Wow, thanks for the tip. Fridge reflection world is so cool. Feels like something David Lynch might dream up. A girl is eating her morning cereal. Suddenly she looks apprehensively at the fridge. Camera dollies towards the appliance and seamlessly penetrates the reflective surface, revealing a deep hidden space that exactly matches the reflection. At the dark end of the tunnel, something stirs... A wildly grinning ma…
Would you be offended if I animated that scene? It is really well described?
Very impressive! Any information on how this compares to 3D Gaussian splatting in terms of performance, quality or data size?
All these details and more in our technical paper! In short: SMERF training takes much longer, SMERF rendering is nearly as fast as 3DGS when a CUDA GPU is available, and quality is visibly higher than 3DGS on large scenes and slightly higher on smaller scenes. https://arxiv.org/abs/2312.07541
Either matterport takes and runs with this or this is a startup waiting to disrupt Realestate.
I can’t believe how smooth this ran on my smartphone.
Feedback: if there was a mode to use the phone compass and gyro for navigation, it’d feel natural. Felt weird to navigate with fingers and figure how to move in xyz dimension.
As others have said, VR mode would be epic.