Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Real-Time Gaussian Splatting

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 61 posts

Re: Show HN: Real-Time Gaussian Splatting

#51

Post by rsp1984 »

markisus wrote: post #36
corysama wrote: post #28

Earlier quoted context omitted.

So, is there some amount of gradient-based optimization going on here? I see RGBD input, transmission, RGBD output. But, other than multi-camera registration, it's difficult to determine what processing took place between input and transmission. What makes this different from RGBD camera visualizations from 10 years ago?

There is no gradient-based optimization. It's (RGBD input, Current Camera Pose) -> Neural Net -> Gaussian Splat output. I'm not aware of other live RGBD visualizations except for direct pointcloud rendering. Compared to pointclouds, splats are better able to render textures, view-dependent effects, and occlusions.

Except that no view-dependent effects that would benefit multi-view consistency are present in your splats.

So yes, it's very much like the RGB-D visualizations from 10 years ago, just with splats instead of points.

Re: Show HN: Real-Time Gaussian Splatting

#53

Post by hi_hi »

While undoubtedly technically impressive, this left me a little confused. Let me explain.

What I think I'm seeing is like one of those social media posts where someone has physically printing out a tweet, taken a photo of them holding the printout, and then posted another social media post of the photo.

Is the video showing me a different camera perspective than what was originally captured, or is this taking a video feed, doing technical magic to convert to gaussian splats, and then converting it back into a (lower quality) video of the same view?

Again, congratulations, this is amazing from a technical perspective, I'm just trying to understand some of the potential applications it might have.

Re: Show HN: Real-Time Gaussian Splatting

#54

Post by markisus »

hi_hi wrote: post #53

While undoubtedly technically impressive, this left me a little confused. Let me explain. What I think I'm seeing is like one of those social media posts where someone has physically printing out a tweet, taken a photo of them holding the printout, and then posted another social media post of the photo. Is the video showing me a different camera perspective than what was originally captured, or is this taking a video…

Yes this converts video stream (plus depth) into Gaussian splats on the fly. While the system is running you can move the camera around to view the splats at different angles.

I took a screen recording of this system as it was running and cut it into clips to make the demo video.

I hope that makes sense?

Re: Show HN: Real-Time Gaussian Splatting

#55

Post by BSVogler »

markisus wrote: post #26
Retr0id wrote: post #18

Earlier quoted context omitted.

How do the view-dependent effects get "discovered" from only a single source camera angle?

Actually there are multiple source cameras. The neural net learns to interpolate the source camera colors based on where the virtual camera is. Under the hood it's hard to say exactly what's going on in the mind of the neural net, but I think it's something like "If I'm closer to camera A, take most of the color from camera A."

Gaussian splatting does not use neural nets. It runs an optimizer on the Gaussian splattering parameters. I think in your comment you are talking about Neural Radiance Fields (Nerfs).

Re: Show HN: Real-Time Gaussian Splatting

#56

Post by Retr0id »

BSVogler wrote: post #55
markisus wrote: post #26

Earlier quoted context omitted.

Actually there are multiple source cameras. The neural net learns to interpolate the source camera colors based on where the virtual camera is. Under the hood it's hard to say exactly what's going on in the mind of the neural net, but I think it's something like "If I'm closer to camera A, take most of the color from camera A."

Gaussian splatting does not use neural nets. It runs an optimizer on the Gaussian splattering parameters. I think in your comment you are talking about Neural Radiance Fields (Nerfs).

Traditionally you'd use an optimizer, but OP isn't doing it traditionally, which is what makes it interesting. NeRFs work differently.

Re: Show HN: Real-Time Gaussian Splatting

#57

Post by markisus »

rsp1984 wrote: post #51
markisus wrote: post #36

Earlier quoted context omitted.

There is no gradient-based optimization. It's (RGBD input, Current Camera Pose) -> Neural Net -> Gaussian Splat output. I'm not aware of other live RGBD visualizations except for direct pointcloud rendering. Compared to pointclouds, splats are better able to render textures, view-dependent effects, and occlusions.

Except that no view-dependent effects that would benefit multi-view consistency are present in your splats. So yes, it's very much like the RGB-D visualizations from 10 years ago, just with splats instead of points.

Here is an example of a view dependent effect produced by LiveSplat [1]. Look closely at the wooden chair handle as the view changes.

I'll concede that ten years ago, someone could have done this. But no one did, as far as I know.

[1] https://imgur.com/a/2yA7eMU

Re: Show HN: Real-Time Gaussian Splatting

#58

Post by markisus »

markisus wrote: post #30
spyder wrote: post #27

Correct me if I'm wrong but looking at the video this just looks like a 3D point cloud using equal-sized "gaussians" (soft spheres) for each pixel, that's why it looks still pixelated especially at the edges. Even when it's low resolution the real gaussian splatting artifacts look different with spikes an soft blobs at the lower resolution parts. So this is not really doing the same as a real gaussian splatting of co…

I had to make a lot of concessions to make this work in real-time. There is no way that I know to replicate the fidelity of "actual" Gaussian splatting training process within the 33ms frame budget. However, I have not baked in the size or orientation into the system. Those are "chosen" by the neural net based on the input RGBD frames. The view dependent effects are also "chosen" by the neural net, but not through an…

I've uploaded a screenshot from LiveSplat where I zoomed in a lot on a piece of fabric. You can see that there is actually a lot of diversity in the shape, orientation, and opacity of the Gaussians produced [1].

[1] https://imgur.com/a/QXxCakM

Post reply on HN