Deep Convolutional Inverse Graphics Network
willwhitney.github.io
Deep Convolutional Inverse Graphics Network
1–10 of 24 posts
Re: Deep Convolutional Inverse Graphics Network
#2In laymen's terms this does what ?
Re: Deep Convolutional Inverse Graphics Network
#3In laymen's terms this does what ?
Network learns a system of lighting and geometry, so you can manipulate a set of codes that represent some variables of that geometry and the positions of those lights.
Re: Deep Convolutional Inverse Graphics Network
#4In laymen's terms this does what ?
Network learns a system of lighting and geometry, so you can manipulate a set of codes that represent some variables of that geometry and the positions of those lights.
In summary, the most interesting part for the general audience might be the following question -- can we learn a 3D rendering engine just from images or videos without any hand-engineering?
Apart from the interesting applications for computer graphics (like rendering novel viewpoints of an object from various viewpoints), this can also be directly used for vision applications. This is because computer vision can be thought of as the inverse of computer graphics.
Goal of computer graphics: scene description -> images
and
Goal of vision: images -> scene description.
Therefore, training a neural network to behave like a graphics engine is interesting from both these perspectives. We are a LONG way from even scratching the surface.
Re: Deep Convolutional Inverse Graphics Network
#5Earlier quoted context omitted.
Network learns a system of lighting and geometry, so you can manipulate a set of codes that represent some variables of that geometry and the positions of those lights.
In summary, the most interesting part for the general audience might be the following question -- can we learn a 3D rendering engine just from images or videos without any hand-engineering? Apart from the interesting applications for computer graphics (like rendering novel viewpoints of an object from various viewpoints), this can also be directly used for vision applications. This is because computer vision can be t…
How long has this idea of making a 3D engine from conv nets been researched?
Re: Deep Convolutional Inverse Graphics Network
#6In laymen's terms this does what ?
It appears that they have developed a novel neural network architecture such that it is possible to understand what each of the layers encode. (Classically, the result of training might not be intelligible--the learned function is encoded in weights and parameters, and aren't meant for human readability or consumption.)
Another interesting property is the "pipeline" and how they seemed to have developed the math to make back propagation work around it. Each step in the pipeline performs some convolution or transformation function.
I haven't read the paper, but I'd be curious to see if they can reuse components of this pipeline in conjunction with one another. Perhaps it wouldn't be immediately possible (I imagine the parameters would have to be adjusted in some shape or form), but a plug-and-play system of pre-trained functions would be nothing sort of amazing.
(I may be incorrect in my analysis. I'm drawing on the ML and image processing I took in undergrad.)
Re: Deep Convolutional Inverse Graphics Network
#7This is very nice, however I wish they would have used a traditional rendering technique (e.g. raytracing) for the decoder stage. It would have been more difficult to compute the gradient, but maybe not too bad if employing some type of automatic differentiation. If they had done it that way, the renderings could scale to any resolution (post-learning) and employ all types of niceities such as depth of field, sub-surface scattering, etc. Instead we're left with these very blocky, quantized convolution-style images.
Re: Deep Convolutional Inverse Graphics Network
#8This is really clever. So basically iiuc, they set up a network to encode down to a representation that consists of parameters for a rendering engine. In order to ensure that this is the representation that is learned, the decoding stage is used to re-render the image subject to transformations and perform the decoding based on a an initial reduction phase after rendering. I.e. it is like an autoencoder, but the inner-most reduced representation is forced to be related to a graphics rendering engine by manipulating related transformation parameters.
Not only is this interesting from the point of view of using it for learning how to generate images, but it is a novel way to force a semantic internal representation instead of leaving it up to a regularisation strategy and interpreting the sparse encoding post-hoc. It forces the internal representation to be inherently "tweakable."
Re: Deep Convolutional Inverse Graphics Network
#9Whoa. Basically like http://www.di.ens.fr/willow/pdfscurrent/pami09a.pdf except it skips the (explicit) 3D mesh reconstruction altogether and goes straight to the rendered output.
Re: Deep Convolutional Inverse Graphics Network
#10Reminds me of being blown away in 2007 by Vetter and Blanz chasing a similar aim: https://m.youtube.com/watch?v=jrutZaYoQJo