Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Real-Time 3D Gaussian Splatting in WebGL

antimatter15.com

Post reply on HN
Print view View original

21–30 of 64 posts

Re: Show HN: Real-Time 3D Gaussian Splatting in WebGL

#21

Post by Lichtso »

porphyra wrote: post #18
Lichtso wrote: post #16

Really cool, I am also working on a port of gaussian-splatting [0] but to WebGPU. Like all the other implementations I have seen so far, this also makes the same mistake when projecting the ellipsoids in a perspective: First you calculate the covariance in 3D and then project that to 2D [1]. This approach only works with parallel / orthographic projections and applying it to perspectives leads to incorrect results. T…

In general, a Gaussian is no longer a true Gaussian after camera projection since the pinhole camera projection function is nonlinear (due to dividing by z). However, if the Gaussian is small relative to the size of the image, you can apporximate it by linearizing the projection function. Therefore the Gaussian splatting paper uses the Jacobian of the projection function as described in equation 5 of the paper [0]. I…

I read the paper and I am aware that the gaussian projection is an approximation anyway (hence I spoke about ellipsoids, not gaussians). Still, one could at least aim to get the iso contour right and yes using the Jacobian matrix is not unsound, just incomplete. As I said, this approach can not produce the distinctive "wiggle" that you get from rotating an ellipsoid while staring dead center at it.

Re: Show HN: Real-Time 3D Gaussian Splatting in WebGL

#22

Post by m1sta_ »

Lichtso wrote: post #16

Really cool, I am also working on a port of gaussian-splatting [0] but to WebGPU. Like all the other implementations I have seen so far, this also makes the same mistake when projecting the ellipsoids in a perspective: First you calculate the covariance in 3D and then project that to 2D [1]. This approach only works with parallel / orthographic projections and applying it to perspectives leads to incorrect results. T…

Dynamic? I've. video?

Re: Show HN: Real-Time 3D Gaussian Splatting in WebGL

#24

Post by naavis »

This is really cool! The control scheme is confusing though. Instead of the typical WASD for moving and using the mouse to look around, dragging the mouse moves forwards and backwards and orbits around some point, A and D strafe, while W and S look up and down.

EDIT: Looks like a full list of controls is in the readme: https://github.com/antimatter15/splat#controls

Re: Show HN: Real-Time 3D Gaussian Splatting in WebGL

#25

Post by contravariant »

Lichtso wrote: post #16

Really cool, I am also working on a port of gaussian-splatting [0] but to WebGPU. Like all the other implementations I have seen so far, this also makes the same mistake when projecting the ellipsoids in a perspective: First you calculate the covariance in 3D and then project that to 2D [1]. This approach only works with parallel / orthographic projections and applying it to perspectives leads to incorrect results. T…

Yeah I think you're right, they're pretending the projection is a linear transformation (in cartesian coordinates) and using it to transform the Gaussian.

Or viewed alternatively they're approximating the projection by assuming all of the Gaussian is at a fixed depth, which I suppose works if it is far enough away.

A projective transformation of a Gaussian seems somewhat annoying, though I assume someone will have done it before. Seems like it should be possible to do it with projective coordinates but the final projection to cartesian coordinates is tricky.

For what it's worth, projecting a contour is also wrong, the whole density changes which also affects the contours.

Re: Show HN: Real-Time 3D Gaussian Splatting in WebGL

#28

Post by porphyra »

Lichtso wrote: post #21
porphyra wrote: post #18

Earlier quoted context omitted.

In general, a Gaussian is no longer a true Gaussian after camera projection since the pinhole camera projection function is nonlinear (due to dividing by z). However, if the Gaussian is small relative to the size of the image, you can apporximate it by linearizing the projection function. Therefore the Gaussian splatting paper uses the Jacobian of the projection function as described in equation 5 of the paper [0]. I…

I read the paper and I am aware that the gaussian projection is an approximation anyway (hence I spoke about ellipsoids, not gaussians). Still, one could at least aim to get the iso contour right and yes using the Jacobian matrix is not unsound, just incomplete. As I said, this approach can not produce the distinctive "wiggle" that you get from rotating an ellipsoid while staring dead center at it.

True, it is an approximation after all. But it is a useful approximation since the main advantage of Gaussian splatting is the speed.

Re: Show HN: Real-Time 3D Gaussian Splatting in WebGL

#30

Post by alanbernstein »

naavis wrote: post #24

This is really cool! The control scheme is confusing though. Instead of the typical WASD for moving and using the mouse to look around, dragging the mouse moves forwards and backwards and orbits around some point, A and D strafe, while W and S look up and down. EDIT: Looks like a full list of controls is in the readme: https://github.com/antimatter15/splat#controls

It's very similar to the N64 FPS controls (i.e. Goldeneye): arrow keys (joystick) for the "primary movements" of forward/backward and yaw, with which you can move and look anywhere in a 2D space. Then, WASD (C buttons) for the "secondary movements" of strafe and pitch.
Post reply on HN