Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DreamFusion: Text-to-3D using 2D Diffusion

dreamfusion3d.github.io

Post reply on HN
Print view View original

111–120 of 208 posts

Re: DreamFusion: Text-to-3D using 2D Diffusion

#111

Post by gamegoblin »

alphabetting wrote: post #91
cuuupid wrote: post #74

Earlier quoted context omitted.

A large portion of the ML community (rightly) discredits Google papers because: - they rarely provide the data or code used so it's basically "i swear it works bro" research - what they achieve is usually through having the most pristine dataset on the planet and is often unusable by other researchers - other times they publish papers that are basically "we slightly modified this excellent open source paper, slapped…

A ton of big advances in AI that community has benefitted from have been from published google research

Notably, the transformer architecture, the basis of the last few years of ML breakthroughs, came out of Google.

Re: DreamFusion: Text-to-3D using 2D Diffusion

#112

Post by thrown_22 »

salawat wrote: post #102
poolio wrote: post #95

Earlier quoted context omitted.

Yes, this is often a problem. We use view-dependent prompts (e.g. "cat wearing sunglasses, back view") but the pretrained 2D model often does not do a good job of interpreting non-canonical views and will put sunglasses on the back of the cats head (as well as the front).

>cat wearing sunglasses, back view") Bad prompt, missing implied antecedent/ambiguous subject... You may want: Back view of a cat which is wearing sunglasses, back view of a cat, but the view is wearing sunglasses, etc... I actually tried using projective terms from drafting books, and didn't get great results. Nor anatomicals either.

>Back view of a cat which is wearing sunglasses, back view of a cat, but the view is wearing sunglasses, etc... I actually tried using projective terms from drafting books, and didn't get great results. Nor anatomicals either.

In short: natural language is not good enough and you need a DSL. If only the last 60 years of language research had warned us of this.

Up next: English sentences are ambiguous and need context information to parse correctly. Machine learning community in shambles.

Re: DreamFusion: Text-to-3D using 2D Diffusion

#113

Post by birracerveza »

In the make-a-video I said that things are getting more and more impressive by the day. I was wrong, because that was a couple hours ago. They're getting more and more impressive by the HOUR.

I'm curious where this will end up in a year. Will it plateau? If so, when?

Re: DreamFusion: Text-to-3D using 2D Diffusion

#114

Post by hwers »

WheelsAtLarge wrote: post #110

I'm not even going to pretend that I have a clue on how this is done. But I'm wondering if the output can be turned into 3d objects that can be used in any of the 3D modeling software? It would be a game changer in terms of real world product development in both of speed and ease.

Well they even have a “downlod model” so yep you definitely can. I wouldn’t think of this as an amazing panacea though, since once everyone has access to it that suddenly means whatever reason making assets like this was valuable before, will now be dirt cheap for all and thus actually net negative for people in that industry. Just saying and warning, not to be a bummer

Re: DreamFusion: Text-to-3D using 2D Diffusion

#115

Post by poolio »

Vt71fcAqt7 wrote: post #99
poolio wrote: post #94

Earlier quoted context omitted.

Co-author here - we were also surprised :) The breadth of knowledge of the visual world embedded in these 2D models and what they unlock is astounding.

Any word about how Nerf -> marching cubes works? I thought that was still an open problem. Is that another discovery in this research paper?

just seems to work when the 3D model is simple and smooth

Re: DreamFusion: Text-to-3D using 2D Diffusion

#116

Post by bmpoole »

salawat wrote: post #102
poolio wrote: post #95

Earlier quoted context omitted.

Yes, this is often a problem. We use view-dependent prompts (e.g. "cat wearing sunglasses, back view") but the pretrained 2D model often does not do a good job of interpreting non-canonical views and will put sunglasses on the back of the cats head (as well as the front).

>cat wearing sunglasses, back view") Bad prompt, missing implied antecedent/ambiguous subject... You may want: Back view of a cat which is wearing sunglasses, back view of a cat, but the view is wearing sunglasses, etc... I actually tried using projective terms from drafting books, and didn't get great results. Nor anatomicals either.

yep our fixed strategy for view-dependent prompting is silly and there is tons of room for improvement!

Re: DreamFusion: Text-to-3D using 2D Diffusion

#117

Post by arkis22 »

poolio wrote: post #94
modeless wrote: post #62

The most incredible thing here is that this demonstrates a level of 3D understanding that I didn't believe existed in 2D image models yet. All of the 3D information in the output was inferred from the training set, which is exclusively uncurated and unsorted 2D still images. No 3D models, no camera parameters, no depth maps. No information about picture content other than a text label (scraped from the web and often…

Co-author here - we were also surprised :) The breadth of knowledge of the visual world embedded in these 2D models and what they unlock is astounding.

the breadth of knowledge of the visual word embedded in these lines and what they unlock is astounding

-the point

Re: DreamFusion: Text-to-3D using 2D Diffusion

#118

Post by amelius »

ToJans wrote: post #38
etaioinshrdlu wrote: post #4

Huh, it's a pretty similar technique to what I outlined a couple days ago: https://news.ycombinator.com/item?id=32965139 Although they start with random initialization and a text prompt. It seems to work well. I now see no reason we can't start with image initialization!

"those who say it cannot be done should not interrupt the people doing it"

They said it could be done, and even said how...

Re: DreamFusion: Text-to-3D using 2D Diffusion

#119

Post by WheelsAtLarge »

hwers wrote: post #114
WheelsAtLarge wrote: post #110

I'm not even going to pretend that I have a clue on how this is done. But I'm wondering if the output can be turned into 3d objects that can be used in any of the 3D modeling software? It would be a game changer in terms of real world product development in both of speed and ease.

Well they even have a “downlod model” so yep you definitely can. I wouldn’t think of this as an amazing panacea though, since once everyone has access to it that suddenly means whatever reason making assets like this was valuable before, will now be dirt cheap for all and thus actually net negative for people in that industry. Just saying and warning, not to be a bummer

Thx, I went back and saw that I missed this the first time:

"Mesh exports Our generated NeRF models can be exported to meshes using the marching cubes algorithm for easy integration into 3D renderers or modeling software."

Like they say, "This is the start of something big."

Re: DreamFusion: Text-to-3D using 2D Diffusion

#120

Post by nbardy »

jianshen wrote: post #14

Did we hit some sort of technical inflection point in the last couple of weeks or is this just coincidence that all of these ML papers around high quality procedural generation are just dropping every other day?

We've hit a couple of inflection point.

The numbers of researchers and research labs scaled up that there is now many well funded teams with experience.

Public tooling and collaboration has reached a point where research happen across the open internet between researchers at a pace that wasn't before possible. Common Crawl, stable diffusion, hugging-face, etc...)

All the techniques that took years in small labs to prove as viable are now getting scaled up across data and people in front of our eyes.

Post reply on HN