Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SPAD: Spatially Aware Multiview Diffusers

yashkant.github.io

Post reply on HN
Print view View original

11–20 of 37 posts

Re: SPAD: Spatially Aware Multiview Diffusers

#11

Post by refulgentis »

bugglebeetle wrote: post #6

I’m confused why there is so much focus on text to images and models. If you spent five minutes talking to anyone with artistic ability, they would tell you that this is not how they generate their work. Making images involves entirely different parts of reasoning than that for speech and language. We seem to be building an entirely faulty model of image generation (outside of things like ControlNet) on the premise t…

Not even wrong, in the Pauli sense: to engage requires ceding the incorrect premises that image models only accept text as input and that the generation process relies on this text

Re: SPAD: Spatially Aware Multiview Diffusers

#12

Post by ummonk »

bugglebeetle wrote: post #6

I’m confused why there is so much focus on text to images and models. If you spent five minutes talking to anyone with artistic ability, they would tell you that this is not how they generate their work. Making images involves entirely different parts of reasoning than that for speech and language. We seem to be building an entirely faulty model of image generation (outside of things like ControlNet) on the premise t…

Project briefs to an artist typically contain both text and reference images. Image diffusion models and the like likewise typically use a text prompt together with optional reference images.

Re: SPAD: Spatially Aware Multiview Diffusers

#13

Post by ilkke »

bugglebeetle wrote: post #6

I’m confused why there is so much focus on text to images and models. If you spent five minutes talking to anyone with artistic ability, they would tell you that this is not how they generate their work. Making images involves entirely different parts of reasoning than that for speech and language. We seem to be building an entirely faulty model of image generation (outside of things like ControlNet) on the premise t…

Check out invoke.ai for an example of something much closer to a professional tool.

Re: SPAD: Spatially Aware Multiview Diffusers

#14

Post by fxtentacle »

Reducing geometric detail while keeping outlines intact is one of the major showstoppers that prevent current game engines from having realistic foliage. And that exact same problem is also why a Nerf with its near-infinite geometric detail is impractical to use for games. And this paper is yet another way to produce a Nerf.

SpeedTree already used billboard textures 10 years ago and that's still the way to go if you need a forest in UE5. Fortnite did slightly improve upon that by having multiple billboard textures that get swapped based on viewing angle, and they call that impostors. But the core issue of how to reduce overdraw and poly count when starting with a high detail object is still unsolved.

That's also the reason, BTW, why UE5's Nanite is used only for mostly solid objects like rocks and statues, but not for trees.

But until this is solved, you always need a technical artist to make a low poly mesh onto whose textures you can bake your high resolution mesh.

Re: SPAD: Spatially Aware Multiview Diffusers

#15

Post by jsheard »

fxtentacle wrote: post #14

Reducing geometric detail while keeping outlines intact is one of the major showstoppers that prevent current game engines from having realistic foliage. And that exact same problem is also why a Nerf with its near-infinite geometric detail is impractical to use for games. And this paper is yet another way to produce a Nerf. SpeedTree already used billboard textures 10 years ago and that's still the way to go if you…

Nanite can actually do trees now, and Fortnite is using it in production, with fully modelled leaves rather than cutout textures because that turned out to be more efficient under Nanite. They talk about it here: https://www.unrealengine.com/en-US/tech-blog/bringing-nanite...

That's still ultimately triangle meshes though, not some other weird representation like NERF, or distance fields, or voxels, or any of the other supposed triangle-killers that didn't stick. Triangles are proving very difficult to kill.

Re: SPAD: Spatially Aware Multiview Diffusers

#16

Post by fxtentacle »

jsheard wrote: post #15
fxtentacle wrote: post #14

Reducing geometric detail while keeping outlines intact is one of the major showstoppers that prevent current game engines from having realistic foliage. And that exact same problem is also why a Nerf with its near-infinite geometric detail is impractical to use for games. And this paper is yet another way to produce a Nerf. SpeedTree already used billboard textures 10 years ago and that's still the way to go if you…

Nanite can actually do trees now, and Fortnite is using it in production, with fully modelled leaves rather than cutout textures because that turned out to be more efficient under Nanite. They talk about it here: https://www.unrealengine.com/en-US/tech-blog/bringing-nanite... That's still ultimately triangle meshes though, not some other weird representation like NERF, or distance fields, or voxels, or any of the oth…

My understanding is that while they allow masked textures, the geometry is still fully emitted by Nanite. That means you still need to start with a mesh that has multiple leaves baked into a single polygon plane, as opposed to starting with individual leaf geometry and then that is somehow baked automatically.

This illustration from the page you linked to shows that as well:

https://cdn2.unrealengine.com/nanite-in-fortnite-chapter-4-p...

The alpha masked holes move around, but the polygons remain static. That means if you draw a tree with this, you still have the full overdraw of the highest-poly mesh.

Re: SPAD: Spatially Aware Multiview Diffusers

#17

Post by jsheard »

fxtentacle wrote: post #16
jsheard wrote: post #15

Earlier quoted context omitted.

Nanite can actually do trees now, and Fortnite is using it in production, with fully modelled leaves rather than cutout textures because that turned out to be more efficient under Nanite. They talk about it here: https://www.unrealengine.com/en-US/tech-blog/bringing-nanite... That's still ultimately triangle meshes though, not some other weird representation like NERF, or distance fields, or voxels, or any of the oth…

My understanding is that while they allow masked textures, the geometry is still fully emitted by Nanite. That means you still need to start with a mesh that has multiple leaves baked into a single polygon plane, as opposed to starting with individual leaf geometry and then that is somehow baked automatically. This illustration from the page you linked to shows that as well: https://cdn2.unrealengine.com/nanite-in-fo…

Yeah alpha masking is inefficient under Nanite, but as they explain further down its handling of dense geometry is good enough that they were able to get away with not using masked materials for the foliage in Fortnite. The individual leaves are modelled as actual geometry and rendered with an opaque, non-masked material.

https://cdn2.unrealengine.com/nanite-in-fortnite-chapter-4-t...

https://cdn2.unrealengine.com/nanite-in-fortnite-chapter-4-t...

Re: SPAD: Spatially Aware Multiview Diffusers

#18

Post by iandanforth »

Please note that these results were obtained using a small amount of compute (compared to say a large language model training run) on a limited training set. Nothing in the paper makes me think that this won't scale. I wouldn't be surprised to see a AAA quality version of this within a few months.
Post reply on HN