Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Real-Time Gaussian Splatting

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 61 posts

Re: Show HN: Real-Time Gaussian Splatting

#33

Post by ttoinou »

markisus wrote: post #26
Retr0id wrote: post #18

Earlier quoted context omitted.

How do the view-dependent effects get "discovered" from only a single source camera angle?

Actually there are multiple source cameras. The neural net learns to interpolate the source camera colors based on where the virtual camera is. Under the hood it's hard to say exactly what's going on in the mind of the neural net, but I think it's something like "If I'm closer to camera A, take most of the color from camera A."

So we’re not sure how it works exactly ?

Re: Show HN: Real-Time Gaussian Splatting

#34

Post by markisus »

whywhywhywhy wrote: post #32

Would be good to see how it's different from just the depth channel applied to the Z of the RGB pixels. Because it looks very similar to that.

The application has this feature and lets you switch back and forth. What you are talking about is the standard pointcloud rendering algorithm. I have an older video where I display the corresponding pointcloud [1] in a small picture in picture frame so you can compare.

I actually started with pointclouds for my VR teleoperation system but I hated how ugly it looked. You end up seeing through objects and objects becoming unparseable if you get too close. Textures present in the RGB frame also become very hard to make out because everything becomes "pointilized". In the linked video you can make out the wood grain direction in the splat rendering, but not in the pointcloud rendering.

[1] https://youtu.be/-u-e8YTt8R8?si=qBjYlvdOsUwAl5_r&t=14

Re: Show HN: Real-Time Gaussian Splatting

#35

Post by markisus »

ttoinou wrote: post #33
markisus wrote: post #26

Earlier quoted context omitted.

Actually there are multiple source cameras. The neural net learns to interpolate the source camera colors based on where the virtual camera is. Under the hood it's hard to say exactly what's going on in the mind of the neural net, but I think it's something like "If I'm closer to camera A, take most of the color from camera A."

So we’re not sure how it works exactly ?

Yup, this is the case for all neural nets.

Re: Show HN: Real-Time Gaussian Splatting

#36

Post by markisus »

corysama wrote: post #28
markisus wrote: post #13

Earlier quoted context omitted.

Gaussian Splatting allows you to create a photorealistic representation of an environment from just a collection of images. Philosophically, this is a form of geometric scene understanding from raw pixels, which has been a holy grail of computer vision since the beginning. Usually creating a Gaussian splat representation takes a long time and uses an iterative gradient-based optimization procedure. Using RGBD helps m…

So, is there some amount of gradient-based optimization going on here? I see RGBD input, transmission, RGBD output. But, other than multi-camera registration, it's difficult to determine what processing took place between input and transmission. What makes this different from RGBD camera visualizations from 10 years ago?

There is no gradient-based optimization. It's (RGBD input, Current Camera Pose) -> Neural Net -> Gaussian Splat output.

I'm not aware of other live RGBD visualizations except for direct pointcloud rendering. Compared to pointclouds, splats are better able to render textures, view-dependent effects, and occlusions.

Re: Show HN: Real-Time Gaussian Splatting

#37

Post by jayd16 »

sendfoods wrote: post #8

Please excuse my naive question - isn't Gaussian Splatting usually used to create 3D imagery from 2D? How does providing 3D input data make sense in this context?

Splatting is about building a scene that supports synthetic view angles.

The depth is helpful to properly handle the parallaxing of the scene as the view angle changes. The system should then ideally "in-paint" the areas that are occluded from the input.

You can either guess the input depth from matching multiple RGB inputs or just use depth inputs along with RGB inputs if you have them. It's not fundamental to the process of building the splats either way.

Re: Show HN: Real-Time Gaussian Splatting

#38

Post by [deleted] »

markisus wrote: post #24
corysama wrote: post #21

So, I see livesplat_realsense.py imports livesplat. Where’s livesplat?

I've tried to make it clear in the link that the actual application is closed source. I'm distributing it as a .whl full of binaries (see the installation instructions). I've considered publishing the source but the source code is is dependent on some proprietary utility libraries from my bigger project and it's hard to fully disentangle it and I'm not sure if this project has some business applications but I'd like…

[deleted]
Post reply on HN