Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DreamFusion: Text-to-3D using 2D Diffusion

dreamfusion3d.github.io

Post reply on HN
Print view View original

131–140 of 208 posts

Re: DreamFusion: Text-to-3D using 2D Diffusion

#131

Post by coolspot »

modeless wrote: post #127
GistNoesis wrote: post #101

Earlier quoted context omitted.

As far as I understand from a quick read of the paper, the 2D diffusion doesn't have a 3D understanding. It probably have some sort of local neighborhood understanding, aka small geometric transformation of objects map close to each other in the diffusion space (That's why like with latent spaces you can "interpolate" ( https://replicate.com/andreasjansson/stable-diffusion-animat... ) in the diffusion space). But tha…

The model clearly has an understanding of the 3D structure of objects. If it didn't, using it to generate 3D models wouldn't work. The knowledge that the leg bone is connected to the knee bone, etc, isn't coming from NeRF, it's all in the "2D" model. Sure, maybe you could distill that knowledge into a different model architecture that is somehow natively 3D in order to improve the efficiency of sampling. But that's m…

> Our eyes only ever see 2D images and we learn 3D structure from that

I think that would be an oversimplification. We do have some 3D information from focus and eye convergence.

Re: DreamFusion: Text-to-3D using 2D Diffusion

#132

Post by ShamelessC »

modeless wrote: post #127
GistNoesis wrote: post #101

Earlier quoted context omitted.

As far as I understand from a quick read of the paper, the 2D diffusion doesn't have a 3D understanding. It probably have some sort of local neighborhood understanding, aka small geometric transformation of objects map close to each other in the diffusion space (That's why like with latent spaces you can "interpolate" ( https://replicate.com/andreasjansson/stable-diffusion-animat... ) in the diffusion space). But tha…

The model clearly has an understanding of the 3D structure of objects. If it didn't, using it to generate 3D models wouldn't work. The knowledge that the leg bone is connected to the knee bone, etc, isn't coming from NeRF, it's all in the "2D" model. Sure, maybe you could distill that knowledge into a different model architecture that is somehow natively 3D in order to improve the efficiency of sampling. But that's m…

The model is trained on discretized photons (pixels). Doing so may have encouraged the model to learn features related to depth.

Re: DreamFusion: Text-to-3D using 2D Diffusion

#133

Post by monocasa »

etaioinshrdlu wrote: post #4

Huh, it's a pretty similar technique to what I outlined a couple days ago: https://news.ycombinator.com/item?id=32965139 Although they start with random initialization and a text prompt. It seems to work well. I now see no reason we can't start with image initialization!

> Sussman attains enlightenment

> In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6.

> “What are you doing?”, asked Minsky.

> “I am training a randomly wired neural net to play Tic-Tac-Toe” Sussman replied.

> “Why is the net wired randomly?”, asked Minsky.

> “I do not want it to have any preconceptions of how to play”, Sussman said.

> Minsky then shut his eyes.

> “Why do you close your eyes?”, Sussman asked his teacher.

> “So that the room will be empty.”

> At that moment, Sussman was enlightened.

http://www.catb.org/jargon/html/koans.html

Re: DreamFusion: Text-to-3D using 2D Diffusion

#134

Post by poolio »

modeless wrote: post #127
GistNoesis wrote: post #101

Earlier quoted context omitted.

As far as I understand from a quick read of the paper, the 2D diffusion doesn't have a 3D understanding. It probably have some sort of local neighborhood understanding, aka small geometric transformation of objects map close to each other in the diffusion space (That's why like with latent spaces you can "interpolate" ( https://replicate.com/andreasjansson/stable-diffusion-animat... ) in the diffusion space). But tha…

The model clearly has an understanding of the 3D structure of objects. If it didn't, using it to generate 3D models wouldn't work. The knowledge that the leg bone is connected to the knee bone, etc, isn't coming from NeRF, it's all in the "2D" model. Sure, maybe you could distill that knowledge into a different model architecture that is somehow natively 3D in order to improve the efficiency of sampling. But that's m…

I think you're both right! It is incredible that the 2D model knows enough about the visual world to produce many objects from all angles, but the 3D model is essential for gluing these views together, and in some ways can fill in the gaps the 2D model doesn't know about. Imagine just taking a huge collection of photographs of an object. While there is enough information in those photos to reconstruct the 3D object, I wouldn't personally call that collection of images "an understanding of 3D." In our case, the diffusion model is the collection of photos and the NeRF model + optimization procedure is what figures out how all those photos can be related to a shared underlying 3D representation. - ben p (author)

Re: DreamFusion: Text-to-3D using 2D Diffusion

#138

Post by bmpoole »

suyash wrote: post #137
bmpoole wrote: post #136

hi folks, ben p from the dreamfusion paper here. happy to answer qs for the next ~hour!

Is there code or notebook for this paper?

not yet, but see the appendix of the paper for pseudocode. the core update step from the diffusion model that powers dreamfusion is surprisingly simple and easy to implement.

Re: DreamFusion: Text-to-3D using 2D Diffusion

#139

Post by visarga »

rileyphone wrote: post #78
WithinReason wrote: post #59

Earlier quoted context omitted.

Isn't that what the Singularity was described as a few decades ago? Progress so fast it's unpredictable even in the short term.

Same as it ever was, scientific revolutions arrive all at once, punctuating otherwise uneventful periods. As I understand, the present one is the product of the paper "Attention is all you need": https://arxiv.org/pdf/1706.03762.pdf .

... that one has 52K citations, and the 2D to 3D paper "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis" with 1488 citations.

https://arxiv.org/abs/2003.08934

Re: DreamFusion: Text-to-3D using 2D Diffusion

#140

Post by visarga »

AStrangeMorrow wrote: post #58
dr_dshiv wrote: post #52

Earlier quoted context omitted.

It’s called the technological singularity. Pretty fun so far!

This isn't what is usually meant by "technological singularity". It is an inflection point where technology growth becomes incontrollable and unpredictable, usually theorized to be cause by a self improving agent (/AI) that becomes smarter with each of its iterations. This is still standard technological progress, human control, even if very fast

It's not really "human controlled". It's an evolutionary process, researchers are scanning the space of possibilities, each with a limited view, but in aggregate it has an emergent positive trend.
Post reply on HN