Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Progressive Growing of GANs for Improved Quality, Stability, Variation [video]

youtube.com

Post reply on HN
Print view View original

61–68 of 68 posts

Re: Progressive Growing of GANs for Improved Quality, Stability, Variation [video]

#61

Post by colllectorof »

AndrewKemendo wrote: post #47
argonaut wrote: post #29

Worth pointing out the main criticism of GANs, which is that right now researchers don't really have a way to tell if a GAN is just copying and pasting the training data or not (there is no "test set" unlike in supervised learning). And in fact an ideal GAN could just learn to output the training set. One example someone found in the generated images for this model: https://twitter.com/nalkalchbrenner/status/92401333…

My interpretation of this "problem" is that it's a radically efficient solution to the problem of generating content in absence of outside reinforcement. Said another way: If we gave a human the task of: [make a painting of a bridge] using a handful of examples of bridges as inspiration, and they did a 1-1 copy of one of them, it would be the most efficient result. However there is generally a culturally implied task…

Can we please stop with these bullshit human comparisons that happen in every AI comment section?

Copying pictures is not efficient within the scope of this problem. The whole point of these algorithms is to extract (or ideally understand) essential features of some class of objects and to be able to represent an object of such class with radically smaller amounts of data that would be required for the full description.

That is the only definition of efficiency that matters here.

Re: Progressive Growing of GANs for Improved Quality, Stability, Variation [video]

#62

Post by colllectorof »

argonaut wrote: post #29

Worth pointing out the main criticism of GANs, which is that right now researchers don't really have a way to tell if a GAN is just copying and pasting the training data or not (there is no "test set" unlike in supervised learning). And in fact an ideal GAN could just learn to output the training set. One example someone found in the generated images for this model: https://twitter.com/nalkalchbrenner/status/92401333…

Yeah. There are a lot of really sketchy transitions that don't look human in that video. Also, when the picture looks fully realistic, you can Google search it and find that there are lots of very similar images.

Now, the fact that they usually aren't identical to Google's finds is moderately impressive. So is the fact that some transitions are "smooth" - being able to move from one head/eye position to another. But the difference between drawing a face and copy-pasting someone's face onto a different hair+background is very significant, and quite often the algorithm seems to be doing the latter. (And in any case, GANs are clearly not the way humans draw faces.)

It would be interesting to see someone try to do the same thing without a neural network. How far would they get on the same training set? The dataset is 30,000 pre-aligned and cropped images. (Would be nice if there was a searchable version to make sure generated versions are not identical to something in that set.)

I bet you could get pretty far with just matching and region replacement, plus some color corrections. But not one would pay you for that.

Re: Progressive Growing of GANs for Improved Quality, Stability, Variation [video]

#63

Post by argonaut »

ericjang wrote: post #56
argonaut wrote: post #29

Worth pointing out the main criticism of GANs, which is that right now researchers don't really have a way to tell if a GAN is just copying and pasting the training data or not (there is no "test set" unlike in supervised learning). And in fact an ideal GAN could just learn to output the training set. One example someone found in the generated images for this model: https://twitter.com/nalkalchbrenner/status/92401333…

This is a general problem for generative models, regardless of whether they are explicit latent density (variational autoencoders) or implicit density (GANs) or fully-observed (PixelCNN). To date no consensus has been reached in the research community on what the "right" way to evaluate a generative model is: e.g. Parzen Windows estimators, log-probability, ELBO, etc. Yann Lecun himself (as of NIPS2016) was pretty cr…

The critic idea seems interesting but doesn't really get to the question of whether/to what degree the GAN is just interpolating the training data. It seems more of a useful tool for diagnosing GANs.

Re: Progressive Growing of GANs for Improved Quality, Stability, Variation [video]

#64

Post by AndrewKemendo »

colllectorof wrote: post #61
AndrewKemendo wrote: post #47

Earlier quoted context omitted.

My interpretation of this "problem" is that it's a radically efficient solution to the problem of generating content in absence of outside reinforcement. Said another way: If we gave a human the task of: [make a painting of a bridge] using a handful of examples of bridges as inspiration, and they did a 1-1 copy of one of them, it would be the most efficient result. However there is generally a culturally implied task…

Can we please stop with these bullshit human comparisons that happen in every AI comment section? Copying pictures is not efficient within the scope of this problem. The whole point of these algorithms is to extract (or ideally understand ) essential features of some class of objects and to be able to represent an object of such class with radically smaller amounts of data that would be required for the full descript…

No we can't. You're right that copying pictures is not the goal - and that is my point. Simply copying pictures would satisfy the "adversarial" side of the DNN inference most efficiently and within the constraints of the GAN architecure. We would consider it a hack or a cheat because the problem is poorly defined, hence the paradox.

Human level AI is the goal (at least mine), so every time we see something unexpected or a "failure" in ML it's worth thinking about the "failure" mode when compared with how a human could hack the system.

Re: Progressive Growing of GANs for Improved Quality, Stability, Variation [video]

#65

Post by alexmlamb2 »

argonaut wrote: post #29

Worth pointing out the main criticism of GANs, which is that right now researchers don't really have a way to tell if a GAN is just copying and pasting the training data or not (there is no "test set" unlike in supervised learning). And in fact an ideal GAN could just learn to output the training set. One example someone found in the generated images for this model: https://twitter.com/nalkalchbrenner/status/92401333…

There are a few things to say about this:

1. If you train a conditional GAN to do image inpainting (for example, left to right), it should be quite apparent the degree to which the model is copying and pasting the training set - by running the model with "given" parts from the test set.

2. I disagree that an ideal GAN could just output the training set. I think the right conceptual framework is that any generative model is trying to produce a distribution similar to the data distribution, and we try to accomplish this by using samples from the data distribution. So if the model memorizes the training set, then it isn't actually that close to the true underlying data distribution. In likelihood-based models (for example the usual generative RNN) you can test this by evaluating likelihood on a validation set.

Re: Progressive Growing of GANs for Improved Quality, Stability, Variation [video]

#66

Post by alexmlamb2 »

AndrewKemendo wrote: post #64
colllectorof wrote: post #61

Earlier quoted context omitted.

Can we please stop with these bullshit human comparisons that happen in every AI comment section? Copying pictures is not efficient within the scope of this problem. The whole point of these algorithms is to extract (or ideally understand ) essential features of some class of objects and to be able to represent an object of such class with radically smaller amounts of data that would be required for the full descript…

No we can't. You're right that copying pictures is not the goal - and that is my point. Simply copying pictures would satisfy the "adversarial" side of the DNN inference most efficiently and within the constraints of the GAN architecure. We would consider it a hack or a cheat because the problem is poorly defined, hence the paradox. Human level AI is the goal (at least mine), so every time we see something unexpected…

To be frank - the problem is not poorly defined, you're just not aware of the definition.

In general in generative models, you have some "true data distribution" P and an estimator distribution Q.

The goal is to make P and Q the same, generally by minimizing some divergence between them.

The actual objective is defined as being between the actual distributions P and Q, but because we only have so many data points, we define an empirical loss that just uses the real observations from P. So if the model makes Q just memorize the samples from P, then it actually hasn't made P and Q similar, it's only minimized the empirical loss.

One practical way to get around this with GANs is to train a conditional GAN instead of an unconditional GAN, and then run the conditioned generation task on held-out samples from the validation set. Another good and perhaps more general solution is to train an inference network and to generate reconstructions on held out data points. If they look totally different, then the model is probably not very "representative".

Re: Progressive Growing of GANs for Improved Quality, Stability, Variation [video]

#67

Post by ericjang »

argonaut wrote: post #63
ericjang wrote: post #56

Earlier quoted context omitted.

This is a general problem for generative models, regardless of whether they are explicit latent density (variational autoencoders) or implicit density (GANs) or fully-observed (PixelCNN). To date no consensus has been reached in the research community on what the "right" way to evaluate a generative model is: e.g. Parzen Windows estimators, log-probability, ELBO, etc. Yann Lecun himself (as of NIPS2016) was pretty cr…

The critic idea seems interesting but doesn't really get to the question of whether/to what degree the GAN is just interpolating the training data. It seems more of a useful tool for diagnosing GANs.

If it were memorizing training data, it would due poorly on a Wasserstein critic trained on a validation set drawn from the same data distribution.

Re: Progressive Growing of GANs for Improved Quality, Stability, Variation [video]

#68

Post by [deleted] »

argonaut wrote: post #29

Worth pointing out the main criticism of GANs, which is that right now researchers don't really have a way to tell if a GAN is just copying and pasting the training data or not (there is no "test set" unlike in supervised learning). And in fact an ideal GAN could just learn to output the training set. One example someone found in the generated images for this model: https://twitter.com/nalkalchbrenner/status/92401333…

[deleted]
Post reply on HN