Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DreamFusion: Text-to-3D using 2D Diffusion

dreamfusion3d.github.io

Post reply on HN
Print view View original

161–170 of 208 posts

Re: DreamFusion: Text-to-3D using 2D Diffusion

#161

Post by deltasevennine »

throwaway675309 wrote: post #56
achr2 wrote: post #51

The thing that frightens me is that we are rapidly reaching broad humanity disrupting ML technologies without any of the social or societal frameworks to cope with it.

I'm usually not a fan of this general hand wringing / fear mongering around ML that a lot of people with too much time and not enough STEM background constantly bring up. Stable diffusion has been made available to the public for quite a while now and if anything has disproved a lot of the ungrounded nonsense that made companies like OpenAI censor their generative models.

>I'm usually not a fan of this general hand wringing / fear mongering around ML that a lot of people with too much time and not enough STEM background constantly bring up.

What's with the STEM reference? Are you implying that STEM is related to intelligence and that people without a STEM background are not intelligent?

It is well known among academics (AKA STEM MAJORS) that human society is a chaotic system and that ML can change society for the better or for the worse... the outcome is basically unknown... It is therefore the intelligent choice to consider the negative consequences of this technology.

To not consider the other side indicates a lack of something.

Re: DreamFusion: Text-to-3D using 2D Diffusion

#164

Post by poolio »

etaioinshrdlu wrote: post #159
bmpoole wrote: post #129

Earlier quoted context omitted.

directly training a NeRF on a single image is a terribly unconstrained problem that would lead to a volume that looks bad when the viewpoint changes. the gist of render + use diffusion model to refine is a great idea though and core to our method! the details of how to use the diffusion model for this refinement was the challenge, but once we figured that out it... just worked :)

Very cool! Can we tweak your algorithm a little to be seeded with a real photo from a single viewpoint?

it's a good idea :)

Re: DreamFusion: Text-to-3D using 2D Diffusion

#165

Post by psychoslave »

modeless wrote: post #127
GistNoesis wrote: post #101

Earlier quoted context omitted.

As far as I understand from a quick read of the paper, the 2D diffusion doesn't have a 3D understanding. It probably have some sort of local neighborhood understanding, aka small geometric transformation of objects map close to each other in the diffusion space (That's why like with latent spaces you can "interpolate" ( https://replicate.com/andreasjansson/stable-diffusion-animat... ) in the diffusion space). But tha…

The model clearly has an understanding of the 3D structure of objects. If it didn't, using it to generate 3D models wouldn't work. The knowledge that the leg bone is connected to the knee bone, etc, isn't coming from NeRF, it's all in the "2D" model. Sure, maybe you could distill that knowledge into a different model architecture that is somehow natively 3D in order to improve the efficiency of sampling. But that's m…

>The model clearly has an understanding of the 3D structure of objects.

And the submarine clearly can swim! :D

The model encompasses 3D information, but a model is not a thinking entity able to understand anything. Well, it might be argued that saying that the "model understands" is a metaphor, and used as such OK, fine. But that’s it. Analogies rarely scale well.

As for humans, have you ever heard of proprioception? If that is not "native 3D sensors", I have no idea of what a 3D sense might be.

Re: DreamFusion: Text-to-3D using 2D Diffusion

#166

Post by F2hP18Foam »

deltasevennine wrote: post #143
coolspot wrote: post #131

Earlier quoted context omitted.

> Our eyes only ever see 2D images and we learn 3D structure from that I think that would be an oversimplification. We do have some 3D information from focus and eye convergence.

its not an over simplification. The extra information from convergence is negligible... our eyes derive virtually identical information when looking at flat 2D pictures of 3D scenes. Evidence of this is everywhere in pictures.

I'm sure people born with one eye are perfectly able to understand the concept of 3D

Re: DreamFusion: Text-to-3D using 2D Diffusion

#167

Post by dbspin »

samstave wrote: post #130

As someone who went to college for 3D animation in +* 1997* + AND DESIGNED the datacenter for luca' presidio complex.. where-by learning that Pixar was developed by steve jobs when lucas didnt think there was a future for computer animation... and so steve bought the death star from lucas... That became pixar... AI is going to fucking kill it - what will happen in the next decade will be ANYONE uploading a script to…

I think this will come, but it won't be competitive with filmmaking in a decade, not without real superhuman AGI. What's much more likely is efforts to reproduce performances will result in deep uncanny valley stuff - and correcting this last few percent for accuracy / weirdness will take a long time. Photorealistic video rendering is inevitable, voice duplication is already here (Tortoise should be much more well known but our culture is fixated on the visual side of this tech - https://colab.research.google.com/drive/1wVVqUPqwiDBUVeWWOUN...). But generatively creating a performance which accurately interprets the emotional context of a scene, in interrelationship with other characters doing the same? I don't see even a path towards that without AGI. At best you could crib specific performance elements and map them onto new models. My intuition is that to get all the way to generative movie you need AGI (or Zimbos so convincing we can't tell the difference).

So what we will get in coming years / decades - the timeline is anyones guess - is movies acted out to camera in a rehearsal space, and that combined with a(n AI augmented) script to generatively create a film / character driven interactive game).

Re: DreamFusion: Text-to-3D using 2D Diffusion

#168

Post by thinkingemote »

bmpoole wrote: post #136

hi folks, ben p from the dreamfusion paper here. happy to answer qs for the next ~hour!

Curious about how long it took: brain storming, research, hypothesis, work, iteration, bug fixing, writing etc

I'm curious about the process you and the team uses to do this. Additionally there's the meme that these things are appearing every hour now, so it could be good for some perspective like "well actually it took n weeks"

Post reply on HN