11–20 of 64 posts
Earlier quoted context omitted.
A thought occurred to me: is this similar to how Media Molecule's Dreams on the PS4/5 renders its scenes?
> Media Molecule “Dreams” has a splat-based renderer (I think the shipped version is not purely splat-based but a combination of several techniques). From: https://aras-p.info/blog/2023/09/05/Gaussian-Splatting-is-pr... Good eye
So far I have only seen gaussian splatting used on photographic data. Would it make sens to use it for other graphics data, too. Or in other words, does it have potential to be used in games?
What am I looking at?
Gaussian splatting is a fancy word for pointcloud but with coloured shapes instead of points. Its been around for ages, but It was never used because if you have a million points in a point cloud, you'd need to artistically manipulate a million points. Its like 3d hair, its pretty simple, just render a billion hairs, but in practice its hard to make it look good. Here we tell a machine learning model to adjust the an…
See the reflections here: https://www.youtube.com/watch?v=mD0oBE9LJTQ
This is also pretty good, but more subtle: https://www.youtube.com/watch?v=tJTbEoxxj0U
What am I looking at?
Gaussian splatting is a fancy word for pointcloud but with coloured shapes instead of points. Its been around for ages, but It was never used because if you have a million points in a point cloud, you'd need to artistically manipulate a million points. Its like 3d hair, its pretty simple, just render a billion hairs, but in practice its hard to make it look good. Here we tell a machine learning model to adjust the an…
Like all the other implementations I have seen so far, this also makes the same mistake when projecting the ellipsoids in a perspective: First you calculate the covariance in 3D and then project that to 2D [1]. This approach only works with parallel / orthographic projections and applying it to perspectives leads to incorrect results. That is because perspective projections have three additional effects:
- Parallax movements (that is the view plane moves parallel to the ellipsoids) change the shape of the projected ellipse. E.g. a sphere only appears circular when in center of the view, once it moves to the edges it becomes stretched into an ellipse. This effect is manually counter balanced by this matrix I believe [2].
- Rotating an ellipse can change the position it appears at, or in other words creates additional translation. This effect is zero if the ellipse has one of its three axes pointing straight at the view (parallel to the normal of the view plane). But, if it is rotated 45°, then the tip of the ellipse that is closer to the view plane becomes larger through the perspective while the other end becomes smaller. Put together, this slightly shifts the center of the appearance away from the projected center of the ellipsoid.
- Conic sections can not only result in ellipses but also parabola and hyperbola. This however is an edge case that only happens when the ellipsoid intersects with the view plane and can probably be ignored as one would clip away such ellipsoids anyway.
The last two effects are not accounted for in these calculations in any of the implementations I have seen so far. What would be correct to do instead? Do not calculate the 3D covariance. Instead calculate the bounding cone around the ellipsoid which has its vertex at the camera position (perspective origin). Then intersect that with the view plane and the resulting conic section is guaranteed to be the correct contour of the perspective projection of the ellipsoid.
[0]: https://github.com/graphdeco-inria/gaussian-splatting [1]: https://github.com/antimatter15/splat/blob/3695c57e8828fedc2... [2]: https://github.com/antimatter15/splat/blob/3695c57e8828fedc2...
Earlier quoted context omitted.
Gaussian splatting is a fancy word for pointcloud but with coloured shapes instead of points. Its been around for ages, but It was never used because if you have a million points in a point cloud, you'd need to artistically manipulate a million points. Its like 3d hair, its pretty simple, just render a billion hairs, but in practice its hard to make it look good. Here we tell a machine learning model to adjust the an…
It's a little bit more than that. Gaussians are view-dependent, which means that they can capture the full radiance field of the scene, rather than just the color and geometry of the objects. All the light bouncing around from different objects can be reproduced, including reflections etc. See the reflections here: https://www.youtube.com/watch?v=mD0oBE9LJTQ This is also pretty good, but more subtle: https://www.yout…
indeed, but that's just adding view dependent points.
Really cool, I am also working on a port of gaussian-splatting [0] but to WebGPU. Like all the other implementations I have seen so far, this also makes the same mistake when projecting the ellipsoids in a perspective: First you calculate the covariance in 3D and then project that to 2D [1]. This approach only works with parallel / orthographic projections and applying it to perspectives leads to incorrect results. T…
[0] https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/3d_...
So far I have only seen gaussian splatting used on photographic data. Would it make sens to use it for other graphics data, too. Or in other words, does it have potential to be used in games?
The industry seems to be trying to move away from this with things like PBR (physical based rendering) and ray / path tracing which enables far better dynamic lighting.
Also, they are extremely space inefficient at the moment. A scene that would take a good traditional rendering engine a few dozen GB would take TB instead. Though, that might improve in the future with more optimization.
One exception to the above, where gaussian splatting might be interesting to see is procedural / generated content (possibly even animated). Especially for volumetric effects which currently use particle systems, like smoke, fire, clouds, flowing water, etc.
Does this use the method proposed by Kerbl and Kopanas at SIGGRAPH 2023? https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/