[flagged]
[deleted]
Show HN: Real-Time Gaussian Splatting
11–20 of 61 posts
Re: Show HN: Real-Time Gaussian Splatting
#12Please excuse my naive question - isn't Gaussian Splatting usually used to create 3D imagery from 2D? How does providing 3D input data make sense in this context?
Well if you have the D channel you might as well benefit from it and have better output
Re: Show HN: Real-Time Gaussian Splatting
#13OP, this is incredible. I worry that people might see a "glitchy 3D video" and might not understand the significance of this. This is getting unreal. They're becoming fast and high fidelity. Once we get better editing capabilities and can shape the Gaussian fields, this will become the prevailing means of creating and distributing media. Turning any source into something 4D volumetric that you can easily mold as clay…
I know enough about 3D rendering to know that Gaussian splatting's one of the Big New Things in high-performance rendering, so I understand that this is a big deal -- but I can't quantify why, or how big a deal it is. Could you or someone else wise in the ways of graphics give me a layperson's rundown of how this works, why it's considered so important, and what the technical challenges are given that an RGB+D(epth?)…
Gaussian Splatting allows you to create a photorealistic representation of an environment from just a collection of images. Philosophically, this is a form of geometric scene understanding from raw pixels, which has been a holy grail of computer vision since the beginning.
Usually creating a Gaussian splat representation takes a long time and uses an iterative gradient-based optimization procedure. Using RGBD helps me sidestep this optimization, as much of the geometry is already present in the depth channel and so it enables the real-time aspect of my technique.
When you say "big deal", I imagine you are also asking about business or societal implications. I can't really speak on those, but I'm open to licensing this IP to any companies which know about big business applications :)
Re: Show HN: Real-Time Gaussian Splatting
#14The output looks terribly similar to what sci-fi movies envisioned as 3D reconstruction of scenes. It is absolutely awesome. Now, if we could project them in 3D… :)
Re: Show HN: Real-Time Gaussian Splatting
#15Another implementation of splat https://github.com/NVlabs/InstantSplat
Re: Show HN: Real-Time Gaussian Splatting
#16Please excuse my naive question - isn't Gaussian Splatting usually used to create 3D imagery from 2D? How does providing 3D input data make sense in this context?
Yes, the normal case uses 2D input, but it can take hours to create the scene. Using the depth channel allows me to create the scene in 33 milliseconds, from scratch, every frame. You could conceptualize this as a compromise between raw pointcloud rendering and fully precomputed Gaussian splat rendering. With pointclouds, you have a lot visual artifacts due to sparsity (low texture information, seeing "through" objects"). With Gaussian splatting, you can transfer a lot more of the 2D texture information into 3D space and render occlusion and view-dependent effects better.
Re: Show HN: Real-Time Gaussian Splatting
#17How did you train this? I'm thinking there isn't reference output for live video frame to splats so supervised learning doesn't work.
Is there some temporal accumulation?
Re: Show HN: Real-Time Gaussian Splatting
#18Please excuse my naive question - isn't Gaussian Splatting usually used to create 3D imagery from 2D? How does providing 3D input data make sense in this context?
Yes, the normal case uses 2D input, but it can take hours to create the scene. Using the depth channel allows me to create the scene in 33 milliseconds, from scratch, every frame. You could conceptualize this as a compromise between raw pointcloud rendering and fully precomputed Gaussian splat rendering. With pointclouds, you have a lot visual artifacts due to sparsity (low texture information, seeing "through" objec…
How do the view-dependent effects get "discovered" from only a single source camera angle?
Re: Show HN: Real-Time Gaussian Splatting
#19How did you train this? I'm thinking there isn't reference output for live video frame to splats so supervised learning doesn't work. Is there some temporal accumulation?
There is no temporal accumulation, but I think that's the next logical step.
Supervised learning actually does work. Suppose you have four cameras. You input the three of them into the net and use the fourth as the ground truth. The live video aspect just emerges from re-running the neural net every frame.
Re: Show HN: Real-Time Gaussian Splatting
#20Another implementation of splat https://github.com/NVlabs/InstantSplat
The quality is better, no doubt, but this method (from the paper) takes on the order of 10-45s depending on input from their table. Which is much better than 10 minutes etc.
That being said, afaict OP's method is 1000x faster, at 33ms.