Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SPAD: Spatially Aware Multiview Diffusers

yashkant.github.io

Post reply on HN
Print view View original

1–10 of 37 posts

Re: SPAD: Spatially Aware Multiview Diffusers

#6

Post by bugglebeetle »

I’m confused why there is so much focus on text to images and models. If you spent five minutes talking to anyone with artistic ability, they would tell you that this is not how they generate their work. Making images involves entirely different parts of reasoning than that for speech and language. We seem to be building an entirely faulty model of image generation (outside of things like ControlNet) on the premise that text and images are equivalent, solely because that’s the training data we have.

Re: SPAD: Spatially Aware Multiview Diffusers

#7

Post by jsheard »

Etherlord87 wrote: post #4
whimsicalism wrote: post #2

No major comment other than this tech is obviously going to transform gaming.

to my understanding it produces 2D images (from various angles), not 3D models… But sure, it's very close to producing a 3D model.

There's been some attempts at translating these techniques over to 3D models but the results so far aren't very useful, they have a tendency to produce extremely triangle-dense meshes which are ironically not actually very detailed or well defined, with most of the detailing just painted over the surface as textures, which also have terrible UV mapping. Not to mention the topology is a mess so it's impossible to rig for animation. I'm not sure how good they are at producing proper PBR material channels, but I'm guessing "not good" if they're derived from text-to-image models trained on fully rendered images/photos rather than individual PBR components, which is much more difficult data to source on a massive scale.

I suspect this will continue to be an uphill battle because there aren't billions of high quality 3D models and PBR textures just lying around on the internet to slurp up and train a model on, so they're having to build it as a second-order system using images as the training data, and muddle though the rest of the steps to get to a usable 3D model.

Re: SPAD: Spatially Aware Multiview Diffusers

#8

Post by Etherlord87 »

jsheard wrote: post #7
Etherlord87 wrote: post #4

Earlier quoted context omitted.

to my understanding it produces 2D images (from various angles), not 3D models… But sure, it's very close to producing a 3D model.

There's been some attempts at translating these techniques over to 3D models but the results so far aren't very useful, they have a tendency to produce extremely triangle-dense meshes which are ironically not actually very detailed or well defined, with most of the detailing just painted over the surface as textures, which also have terrible UV mapping. Not to mention the topology is a mess so it's impossible to rig…

I'm looking at such techniques that appear here on HN every now and then, and indeed, they are very impressive, but also something an intermediate 3D artist will do better with ease.

I think however, that the existence of such tools could perhaps motivate more people to create low quality 3D assets, that for many purposes are good enough - but people might be shy to do that due to many high quality assets shaming them... Once AI 3D stuff will flood the space, amateurs might start competing with it - or so I hope.

Also it's just a matter of time until we reach the 80% (from the 80/20 rule), 90% and 98% quality of these tools. The remaining 2% will still be of value in AAA titles though...

Re: SPAD: Spatially Aware Multiview Diffusers

#9

Post by teaearlgraycold »

Teknomancer wrote: post #3
whimsicalism wrote: post #2

No major comment other than this tech is obviously going to transform gaming.

My immediate thoughts as well, but I wonder about how exactly this could be implemented in current game build chains such as unreal or unity?

I expect people will implement rough geometry with polygons and feed a depth+material map to a diffuser for rasterization. This way you get photorealism from the models but maintain precise control over what the scene has in it and where.

Re: SPAD: Spatially Aware Multiview Diffusers

#10

Post by teaearlgraycold »

bugglebeetle wrote: post #6

I’m confused why there is so much focus on text to images and models. If you spent five minutes talking to anyone with artistic ability, they would tell you that this is not how they generate their work. Making images involves entirely different parts of reasoning than that for speech and language. We seem to be building an entirely faulty model of image generation (outside of things like ControlNet) on the premise t…

It’s good for stock images.

And for in-painting I think you’ll find text-to-image is still useful to artists. It’s extra metadata to guide the generation of a small portion of the final image.

Post reply on HN