Live data from Hacker News

Triangle splatting: radiance fields represented by triangles

trianglesplatting.github.io

51–60 of 81 posts

Re: Triangle splatting: radiance fields represented by triangles

#51
post #9

Earlier quoted context omitted.

And in case it helps further in the context of the article: traditional rendering pipelines for games don't render fuzzy Gaussian points, but triangles instead. Having the model trained on how to construct triangles (rather than blobbly points) means that we're closer to a "take photos of a scene, process them automatically, and walk around them in a game engine" style pipeline.

Any insights into why game engines prefer triangles rather than guassians for fast rendering? Are triangles cheaper for the rasterizer, antialiasing, or something similar?

As an aside, a few early 90s games did experiment with spheroid sprites to approximate 3D rendering, including the DOS game Ecstatica [1] and the (unfortunately named) SNES/Genesis game Ballz 3D [2]

[1] https://www.youtube.com/watch?v=nVNxnlgYOyk

[2] https://www.youtube.com/watch?v=JfhiGHM0AoE

Re: Triangle splatting: radiance fields represented by triangles

#52

This looks really nice, but I cant help to think this is a stop gap solution like other splatting techniques. It's certainly better than NERFs, where the whole scene is contained in a black box, but reality is not made up of a triangle soup or gaussian blobs. Most of the real world is made up of volumes, but can often be thought of as surfaces. It makes sense to represent the ground, a table, walls, etc with planes,…

A digital image is a soup, of RGB dots of various size. Gaussian Splatting radically changed the approach to photogrammetry. Prior approaches to generate surface models, and mapping the captures to materials that a renderer would more or less rasterize with physically accuracy were hitting the ceiling of the technique. NerF was also a revolution but is very compute intensive. Even a browser, a mid range GPU, can rend…

No, an image is a well ordered grid of pixels. The 3D variant would be voxels, and Nvidia recently released a project to do scene reconstruction with sparse voxels [0].

If you take these triangles, make them share vertices, and order them in a certain way, you have a mesh. You can then combine some of them into larger flat surfaces when that makes sense, draw thousands of them in one draw call, calculate intersections, volumes, physics, LODs, use textures with image compression instead of millions of colored objects, etc with them. Splatting is one way of answering the question "how do we reproduce these images in a way that lets us generate novel views of the same scene", not "what is the best representation of this 3D scene".

The aim is to find the light field that describes the scene, and if you have solid objects that function can be described on the surface of those objects. Seems like a much more elegant end result than a cloud of separate objects, no matter what shape they have, since that's much closer to how reality works. Obviously we need to handle volumetrics and translucency as well, but if we model the real surfaces as virtual surfaces I think things like reflections and shadow removal will be easier. At least gaussian splats have a hard time with reflections, they look good from some viewing angles, but the reflections are often handled as geometry [1].

I'm not arguing that it doesn't look good or that it doesn't serve a purpose, sometimes a photorealistic novel view of a real scene is all you want. But I still don't think it's the best representation of scenes.

[0] https://svraster.github.io/

[1] https://www.youtube.com/watch?v=yq6gtdpLUCo

Re: Triangle splatting: radiance fields represented by triangles

#53
post #9

Earlier quoted context omitted.

And in case it helps further in the context of the article: traditional rendering pipelines for games don't render fuzzy Gaussian points, but triangles instead. Having the model trained on how to construct triangles (rather than blobbly points) means that we're closer to a "take photos of a scene, process them automatically, and walk around them in a game engine" style pipeline.

Any insights into why game engines prefer triangles rather than guassians for fast rendering? Are triangles cheaper for the rasterizer, antialiasing, or something similar?

>triangles cheaper for the rasterizer

Yes, using triangles simplifies a lot of math, and GPUs were created to be really good at doing the math related to triangles rasterization (affine transformations).

Re: Triangle splatting: radiance fields represented by triangles

#54
Can anyone detail the use case for gaussian splatting to me? What are we trying to solve, or, where direction are we trying to head towards?

I'm more familiar with traditional 3D graphics, so this new wave of papers around gaussian splatting lies outside my wheelhouse.

Re: Triangle splatting: radiance fields represented by triangles

#55

Can anyone detail the use case for gaussian splatting to me? What are we trying to solve, or, where direction are we trying to head towards? I'm more familiar with traditional 3D graphics, so this new wave of papers around gaussian splatting lies outside my wheelhouse.

I get the impression the goal is to save 3D environments with baked lighting without having to run raytracing, at a level above explicitly defined meshes with faces covered by 2D textures, which can't represent fog, translucency, reflection glints, etc without a separate lighting pass. Basically trying to get raytracing without doing raytracing.

Re: Triangle splatting: radiance fields represented by triangles

#56

Can anyone detail the use case for gaussian splatting to me? What are we trying to solve, or, where direction are we trying to head towards? I'm more familiar with traditional 3D graphics, so this new wave of papers around gaussian splatting lies outside my wheelhouse.

AFAIK Gaussian Splatting is somehow connected to NeRFs (neural radiance fields), so job of turning multiple 2D images into 3D scene. Actually tried doing something like this recently for drone navigation (using older point cloud methods) but no luck so far.

Can anyone who read this suggest something to use to scan room geometry using camera only in real-time (with access to beefy NVIDIA computer if needed) for drone navigation purposes?

Re: Triangle splatting: radiance fields represented by triangles

#58
This seems like the natural next step after Gaussian splatting. After all, triangles are pretty much the most "native" rendering that GPUs can do. And as long as you figure out a way to make it differentiable (e.g. with their windowing function), it should be possible to just throw your triangles into a big optimizer.

Re: Triangle splatting: radiance fields represented by triangles

#59

Can anyone detail the use case for gaussian splatting to me? What are we trying to solve, or, where direction are we trying to head towards? I'm more familiar with traditional 3D graphics, so this new wave of papers around gaussian splatting lies outside my wheelhouse.

I get the impression the goal is to save 3D environments with baked lighting without having to run raytracing, at a level above explicitly defined meshes with faces covered by 2D textures, which can't represent fog, translucency, reflection glints, etc without a separate lighting pass. Basically trying to get raytracing without doing raytracing.

I would say they’re an attempt to extend the concept of a photograph in to truly 3 dimensions (not a 2D bitmap with a depth layer)

Re: Triangle splatting: radiance fields represented by triangles

#60

Can anyone detail the use case for gaussian splatting to me? What are we trying to solve, or, where direction are we trying to head towards? I'm more familiar with traditional 3D graphics, so this new wave of papers around gaussian splatting lies outside my wheelhouse.

Gaussian splatting models the scene as a bunch of normal distributions (fuzzy squished spheres) instead of triangles, then renders those with billboarded triangles. It has advantages (simpler representation, easy to automatically capture from a scan) and disadvantages (not what the hardware is designed for, not watertight). The biggest disadvantage is that most graphics techniques need to be reinvented for it, and it's not clear what the full list of advantages and disadvantages will be until people have done all of those. But that big disadvantage is also a great reason to make tons of papers.
Post reply on HN