Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Convolutional Inverse Graphics Network

willwhitney.github.io

Post reply on HN
Print view View original

11–20 of 24 posts

Re: Deep Convolutional Inverse Graphics Network

#11

Post by tejask »

svantana wrote: post #7

This is very nice, however I wish they would have used a traditional rendering technique (e.g. raytracing) for the decoder stage. It would have been more difficult to compute the gradient, but maybe not too bad if employing some type of automatic differentiation. If they had done it that way, the renderings could scale to any resolution (post-learning) and employ all types of niceities such as depth of field, sub-sur…

One of the authors here. You are absolutely right! In fact, I am currently doing something similar but it is not working as well yet. As far as this work is concerned, we wanted to see how model-free can we go.

Re: Deep Convolutional Inverse Graphics Network

#12

Post by tejask »

perdunov wrote: post #5
tejask wrote: post #4

Earlier quoted context omitted.

In summary, the most interesting part for the general audience might be the following question -- can we learn a 3D rendering engine just from images or videos without any hand-engineering? Apart from the interesting applications for computer graphics (like rendering novel viewpoints of an object from various viewpoints), this can also be directly used for vision applications. This is because computer vision can be t…

How long has this idea of making a 3D engine from conv nets been researched?

To the best of my knowledge, not much at all. It is an open question. Besides, a feedforward net is not going to be enough.

Re: Deep Convolutional Inverse Graphics Network

#13

Post by rndn »

tejask wrote: post #11
svantana wrote: post #7

This is very nice, however I wish they would have used a traditional rendering technique (e.g. raytracing) for the decoder stage. It would have been more difficult to compute the gradient, but maybe not too bad if employing some type of automatic differentiation. If they had done it that way, the renderings could scale to any resolution (post-learning) and employ all types of niceities such as depth of field, sub-sur…

One of the authors here. You are absolutely right! In fact, I am currently doing something similar but it is not working as well yet. As far as this work is concerned, we wanted to see how model-free can we go.

I don’t understand much of the paper but it looks awesome! I have two questions: Am I understanding it correctly that one would need to convert the internal representation to a textured triangle mesh in order to use ray tracing in the decoder stage? Is the encoder effectively similar to scene reconstruction via structure from motion?

Re: Deep Convolutional Inverse Graphics Network

#14

Post by amelius »

So, in essence, this network can learn to "unproject" images.

Since projection is a lossy operation, a projected image has potentially multiple inverses. And this makes me wonder how this system deals with the situation where two or more inverses exist and are equally likely.

Re: Deep Convolutional Inverse Graphics Network

#15

Post by tejask »

rndn wrote: post #13
tejask wrote: post #11

Earlier quoted context omitted.

One of the authors here. You are absolutely right! In fact, I am currently doing something similar but it is not working as well yet. As far as this work is concerned, we wanted to see how model-free can we go.

I don’t understand much of the paper but it looks awesome! I have two questions: Am I understanding it correctly that one would need to convert the internal representation to a textured triangle mesh in order to use ray tracing in the decoder stage? Is the encoder effectively similar to scene reconstruction via structure from motion?

there are many ways to parametrize the decoder. One of the ways is to constrain it to output an explicit mesh or volumetric representation and express the rendering pipeline so that it's differentiable. The encoder will then effectively learn an "inference algorithm" to get the best output. A feedforward neural network is not enough and recurrent computations will eventually be necessary.

Re: Deep Convolutional Inverse Graphics Network

#17

Post by rirarobo »

Very cool work, I'm happy to see more people thinking about deep networks along these lines. It seems that this is very similar to a recent work put on arxiv back in November,

"Learning to Generate Chairs with Convolutional Neural Networks". http://arxiv.org/abs/1411.5928

They also have a very cool video of the generation process: https://youtu.be/QCSW4isBDL0

It's very interesting to see two groups independently developing almost identical networks for inverse graphics tasks, both using pose, shape, and view parameters to guide learning. I think that continuing in this direction could provide a lot of insight into how these deep networks work, and lead to new improvements for recognition tasks too.

@tejask - You should probably cite the above paper, and thanks for providing code! awesome!

Re: Deep Convolutional Inverse Graphics Network

#19

Post by cs702 »

radarsat1 wrote: post #8

This is really clever. So basically iiuc, they set up a network to encode down to a representation that consists of parameters for a rendering engine. In order to ensure that this is the representation that is learned, the decoding stage is used to re-render the image subject to transformations and perform the decoding based on a an initial reduction phase after rendering. I.e. it is like an autoencoder, but the inne…

This can also be used for object recognition against invariant 3D representations, potentially with more accuracy than traditional convolutional neural net architectures.

Consider: their proof-of-concept face-recognition model achieves performance comparable to traditional convnets on faces with varying degree of pose, lighting, shape and texture, even though it was trained completely unsupervised. I would expect this type of model to beat the state of the art in face recognition and other similar tasks when fined-tuned with supervised training in the not-too-distant future.

Re: Deep Convolutional Inverse Graphics Network

#20

Post by jwp729 »

tejask wrote: post #15
rndn wrote: post #13

Earlier quoted context omitted.

I don’t understand much of the paper but it looks awesome! I have two questions: Am I understanding it correctly that one would need to convert the internal representation to a textured triangle mesh in order to use ray tracing in the decoder stage? Is the encoder effectively similar to scene reconstruction via structure from motion?

there are many ways to parametrize the decoder. One of the ways is to constrain it to output an explicit mesh or volumetric representation and express the rendering pipeline so that it's differentiable. The encoder will then effectively learn an "inference algorithm" to get the best output. A feedforward neural network is not enough and recurrent computations will eventually be necessary.

Can you explain a bit more why the recurrent network structure becomes necessary at some point? Is that because reversing a CNN naturally means rendering by (de)convolution?
Post reply on HN