Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DreamFusion: Text-to-3D using 2D Diffusion

dreamfusion3d.github.io

Post reply on HN
Print view View original

171–180 of 208 posts

Re: DreamFusion: Text-to-3D using 2D Diffusion

#171

Post by lolspace »

deltasevennine wrote: post #161
throwaway675309 wrote: post #56

Earlier quoted context omitted.

I'm usually not a fan of this general hand wringing / fear mongering around ML that a lot of people with too much time and not enough STEM background constantly bring up. Stable diffusion has been made available to the public for quite a while now and if anything has disproved a lot of the ungrounded nonsense that made companies like OpenAI censor their generative models.

>I'm usually not a fan of this general hand wringing / fear mongering around ML that a lot of people with too much time and not enough STEM background constantly bring up. What's with the STEM reference? Are you implying that STEM is related to intelligence and that people without a STEM background are not intelligent? It is well known among academics (AKA STEM MAJORS) that human society is a chaotic system and that…

That's exactly what he's saying. How many citations does the top cited paper from the LessWrong community have?

Re: DreamFusion: Text-to-3D using 2D Diffusion

#172

Post by hjaarnio »

poolio wrote: post #157
LarsDu88 wrote: post #154

As someone who dabbles in 3d modeling, this is going to be an incredible resource for creating static 3d objects. Someone ought to come up with a way to convert to mesh better than the Marching Cubes algorithm I've seen applied to most NERFs. The models still lack coherent topology and would probably be janky if fully rigged.

With smooth enough geometry converting NeRFs to meshes with marching cubes works pretty well. Would you say the topology of meshes on our website are still too incoherent for rigging?

In case of a communication gap; The word 'topology' has a more domain specific meaning in animation and rigging compared to the mathematical one.

It's used to mean that the placement of the lower level components - vertices, edges and faces - are well aligned to the higher level structure of the object, and make up a well defined 2D grid that flows along the models surface. In particular you'd want edges going along/perpendicular to mesh structures such as limbs, and around facial features and other details in a logical manner.

Otherwise when applying deformations as part of an animation the model will not have the detail in the right places to still look good, e.g. if there is no edges perpendicular to a joint in a limb, the bent version of the limb cannot have a clear smooth line along the joint, and the edges and faces become janky.

Under this definition, marching cubes cannot produce good 2D topology, as the mesh features are all aligned to the cardinal grid instead of the features of the object represented.

Re: DreamFusion: Text-to-3D using 2D Diffusion

#173

Post by poolio »

thinkingemote wrote: post #168
bmpoole wrote: post #136

hi folks, ben p from the dreamfusion paper here. happy to answer qs for the next ~hour!

Curious about how long it took: brain storming, research, hypothesis, work, iteration, bug fixing, writing etc I'm curious about the process you and the team uses to do this. Additionally there's the meme that these things are appearing every hour now, so it could be good for some perspective like "well actually it took n weeks"

Great question! Our team has been working on text-to-3d for ~1.5 years starting with https://ajayj.com/dreamfields. We had hoped that we could swap the contrastive CLIP model in Dream Fields for the generative Imagen model and crank out an easy paper in a few weeks. But what was supposed to be an easy win turned into months of frustration. Nothing we tried worked any better than Dream Fields. After a long detour trying MCMC, we stumbled across the score distillation loss that powers DreamFusion. Going from an initial sign of life to the results you see today still took months of hard work.

Research progress is unpredictable and these advances are not inevitable. We have the privilege to work in an environment full of amazing colleagues and powerful models, but at the end of the day it took a persistent team and a bit of luck.

Re: DreamFusion: Text-to-3D using 2D Diffusion

#174

Post by poolio »

hjaarnio wrote: post #172
poolio wrote: post #157

Earlier quoted context omitted.

With smooth enough geometry converting NeRFs to meshes with marching cubes works pretty well. Would you say the topology of meshes on our website are still too incoherent for rigging?

In case of a communication gap; The word 'topology' has a more domain specific meaning in animation and rigging compared to the mathematical one. It's used to mean that the placement of the lower level components - vertices, edges and faces - are well aligned to the higher level structure of the object, and make up a well defined 2D grid that flows along the models surface. In particular you'd want edges going along/…

Aha thank you, this is helpful. Agreed there is much research needed to get this working but hopefully not too far off: https://twitter.com/kkpatain/status/1575758085821706240

Re: DreamFusion: Text-to-3D using 2D Diffusion

#175

Post by poolio »

bertdb wrote: post #169
bmpoole wrote: post #136

hi folks, ben p from the dreamfusion paper here. happy to answer qs for the next ~hour!

Could the NeRF be replaced with a voxel grid, backpropagating to the voxel color values directly? Or is there a reason that wouldn't work?

should work, and there are tons of new differentiable mesh and volumetric representations to try!

Re: DreamFusion: Text-to-3D using 2D Diffusion

#176

Post by rjmunro »

drKarl wrote: post #40
drKarl wrote: post #25

Amazing! How long then until we get photorealistic AI generated 3D VR games and experiences in the metaverse?

Why the downvote? I wasn't being sarcastic, it was a honest question, I'm really impressed how far this technology has come since GPT-3 2 years ago to DALl-E and Stable Diffusion ro Meta's text to video to this...

Maybe because you said "Metaverse" (and to some extent "VR") making it sound like sci-fi nonsense. You could have just said:

How long then until we get photorealistic AI generated 3D games and experiences?

Re: DreamFusion: Text-to-3D using 2D Diffusion

#177

Post by machina_ex_deus »

beambot wrote: post #57
layer8 wrote: post #22

Earlier quoted context omitted.

From the abstract: “We introduce a loss based on probability density distillation that enables the use of a 2D diffusion model as a prior for optimization of a parametric image generator. Using this loss in a DeepDream-like procedure, we optimize a randomly-initialized 3D model (a Neural Radiance Field, or NeRF) via gradient descent such that its 2D renderings from random angles achieve a low loss.” This seems like b…

> This seems like basically plugging a couple of techniques together that already existed [...] In his Lex Fridman interview, John Carmack makes similar assertions about this prospect for AGI: That it will likely be the clever combination of existing primitives (plus maybe a couple novel new ones) that make the first AGI feasible in just a couple thousand lines of code.

Billions of creatures with stronger neutral networks, more parameters, better input have lived on earth for millions of years, but only now something like humans showed up. I fully expect AI to do everything animals can do pretty soon, but since whatever it is that differentiates humans didn't happen for million of years, there's good chance AGI research will get stuck at a similar point.

Re: DreamFusion: Text-to-3D using 2D Diffusion

#178

Post by anentropic »

adamredwoods wrote: post #66
modeless wrote: post #62

The most incredible thing here is that this demonstrates a level of 3D understanding that I didn't believe existed in 2D image models yet. All of the 3D information in the output was inferred from the training set, which is exclusively uncurated and unsorted 2D still images. No 3D models, no camera parameters, no depth maps. No information about picture content other than a text label (scraped from the web and often…

So I wonder if unusual angles that normally do not get photographed will be distorted? For example, underneath a table looking up.

You can see that in some of these examples e.g. "plush toy of a corgi nurse"
Post reply on HN