Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DreamFusion: Text-to-3D using 2D Diffusion

dreamfusion3d.github.io

Post reply on HN
Print view View original

101–110 of 208 posts

Re: DreamFusion: Text-to-3D using 2D Diffusion

#101

Post by GistNoesis »

modeless wrote: post #62

The most incredible thing here is that this demonstrates a level of 3D understanding that I didn't believe existed in 2D image models yet. All of the 3D information in the output was inferred from the training set, which is exclusively uncurated and unsorted 2D still images. No 3D models, no camera parameters, no depth maps. No information about picture content other than a text label (scraped from the web and often…

As far as I understand from a quick read of the paper, the 2D diffusion doesn't have a 3D understanding. It probably have some sort of local neighborhood understanding, aka small geometric transformation of objects map close to each other in the diffusion space (That's why like with latent spaces you can "interpolate" (https://replicate.com/andreasjansson/stable-diffusion-animat...) in the diffusion space).

But that's not really surprising because when you have enough data, even simple clustering methods group objects like faces by the direction they are looking to. With enough views even a simple L2 distance in pixel space allow t-SNE to do that.

They are injecting the 3D constraints via the NERF and an optimization process to add the consistency between the frames.

It's a deep dream process that optimize by alternating updates for 3D consistency, and updates for text-to-2Dimage correspondence. It's searching for a solution that satisfy these two constraints at the same time.

Even though they only need to run a single diffusion step to get the update direction, this optimization process is quite long : 1h30 (but they are not using things like instant Nerf (or even simple voxel grids) ).

But this will allow for creation of a dataset of 3D objects with corresponding text, which will then allow to train a diffusion model that will have a 3D understanding and will be able to generate 3D objects directly with a single diffusion process.

Re: DreamFusion: Text-to-3D using 2D Diffusion

#102

Post by salawat »

poolio wrote: post #95
adamredwoods wrote: post #66

Earlier quoted context omitted.

So I wonder if unusual angles that normally do not get photographed will be distorted? For example, underneath a table looking up.

Yes, this is often a problem. We use view-dependent prompts (e.g. "cat wearing sunglasses, back view") but the pretrained 2D model often does not do a good job of interpreting non-canonical views and will put sunglasses on the back of the cats head (as well as the front).

>cat wearing sunglasses, back view")

Bad prompt, missing implied antecedent/ambiguous subject...

You may want:

Back view of a cat which is wearing sunglasses, back view of a cat, but the view is wearing sunglasses, etc... I actually tried using projective terms from drafting books, and didn't get great results. Nor anatomicals either.

Re: DreamFusion: Text-to-3D using 2D Diffusion

#103

Post by the8472 »

jianshen wrote: post #14

Did we hit some sort of technical inflection point in the last couple of weeks or is this just coincidence that all of these ML papers around high quality procedural generation are just dropping every other day?

This has been going on for years. The applications are just crossing thresholds now that are more salient for people, e.g. doing art.

Re: DreamFusion: Text-to-3D using 2D Diffusion

#106

Post by rirarobo »

O__________O wrote: post #41
parasj wrote: post #35

Earlier quoted context omitted.

This link was from OpenReview which must be anonymous (double blind). The full author list is on the updated link at: https://dreamfusion3d.github.io

Aware of the link, though you have not provided any clarification for why there are two links; strikes me as odd if authors are trying to post it anonymously that simple Google finds authors names.

I believe ICLR guidelines require the authors to submit papers and any supplementary materials (including links to webpages, videos, etc) without identifying information, but authors are not barred from public announcements on other forums. IIUC, the idea behind this policy was originally to accommodate author freedom to engage in common practices such as simultaneous submission to arxiv (which identifies the authors). To respect the double blind spirit of review, reviewers are asked not to actively search the web in attempt identify the authors. In the past, when social media promotion was less common, it was reasonably likely that reviewers would follow this guidance and would not have seen the arxiv submission, preserving the double blind nature of review in most cases. However, the use of social media in academia has radically changed in recent years, as more researchers use social media to keep up with the latest advancements, so promotion of papers in submission on platforms like Twitter can offer significant advantages to authors.

So, authors today often submit anonymously following the conference guidelines, but simultaneously post publicly elsewhere, walking a fine line as not to overstep the conference policies. This appears to be the case for this submission. Note, recently, some conferences, such as CVPR, have started to institute new policies forbidding social media promotion until acceptance, as they adapt to the changing landscape of social media promotion. If this were a CVPR submission, the authors would not be allowed to tweet publicly about their work yet, nor have the version of the webpage with their names visible.

Re: DreamFusion: Text-to-3D using 2D Diffusion

#107

Post by ecmascript »

LordDragonfang wrote: post #100
samstave wrote: post #46

Earlier quoted context omitted.

Scary fn thought! and I agree with you! And the OP comment its by the magnanimous/infamous AnigBrowl You need to start doing AI legal admin ( I dont have the terms, but you may - we need legal language to control how we deal with AI) and @dang - kill the gosh darn "posting too fast" thing Jiminey Crickets I have talked to you abt this so many times...

"You're posting too fast" is a limit that's manually applied to accounts that have a history of "posting too many low-quality comments too quickly and/or getting into flame wars". You can email dang (hn@ycombinator.com) if you think it was applied in error, but if it's been applied to you more than once... you probably have a continuing problem with proliferating flame wars or posting otherwise combative comments.

I think you also can get it by just having unpopular opinions. Hackernews used to be much more nuanced than it is today imo.

Re: DreamFusion: Text-to-3D using 2D Diffusion

#108

Post by sirianth »

jaggs wrote: post #43
sirianth wrote: post #13

Earlier quoted context omitted.

Silly replying to myself I know, but I had more thoughts. I'm an architect for 3D worlds and I am desperate, lol, for this kind of tool. I use both blender and grasshopper, but I use midjourney to think and prototype all the time. Obvious but it would be astonishing to have something like this for game worlds. I used another version of this to create "a forest emerging from an aircraft carrier" https://www.instagram.…

You should try the AUTOMATIC1111 version of stable diffusion. It's crazy fast and has great results - https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki...

I'm mostly only looking for 3DML these days. I want it to hallucinate architecture for games.

Re: DreamFusion: Text-to-3D using 2D Diffusion

#110

Post by WheelsAtLarge »

I'm not even going to pretend that I have a clue on how this is done. But I'm wondering if the output can be turned into 3d objects that can be used in any of the 3D modeling software? It would be a game changer in terms of real world product development in both of speed and ease.
Post reply on HN