Live data from Hacker News

Text to Image Synthesis Using Thought Vectors

github.com

21–23 of 23 posts

Re: Text to Image Synthesis Using Thought Vectors

#21
post #17

I think the idea is interesting but I'm not convinced it really "synthesizes ideas" so much as treats the neural network like a database of images that it mixes. Now, I could be wrong, but because of the way the results are presented it doesn't tell me that it's any good at picking up the meaning of the phrase. The results show a single phrase and a set of images it generates. White flower with yellow center, and a b…

How do you "synthesize ideas" if not by combining parts of your own personal database of images/concepts? Even in your example of "X flower with Y center [and Z features]" you start with a mental picture of a flower you've seen (or a generalisation from many flowers you've seen) and then modify it with your mental picture of colours X and Y and features Z.

>How do you "synthesize ideas" if not by combining parts of your own personal database of images/concepts?

Procedural generation can be far more complex than just linear blending, which is all that a shallow net can do. For example, consider the full generative process which creates a frame from the game No Man's Sky. It is enormously more complex than a simple shallow net that can just do linear blends of previous examples - many many nonlinear processing steps to go from a small random seed to intermediate databases for terrain and objects and finally down to pixels.

If you look at the actual net design used here, it's only a few layers deep, and not very big. Much much closer to 'linear blending' than what our brains do (which is presumably vaguely closer to what no man's sky does).

Re: Text to Image Synthesis Using Thought Vectors

#22
post #18
post #4

Earlier quoted context omitted.

There is still a long way to go, to be able to do that. The model currently generates 64 X 64 pictures and is trained on a very specific flowers image dataset. Nevertheless, it would be a great idea to experiment with such a dataset (of sketches and descriptions) if available.

But the thing about machine learning is that once it works at all, "a long way to go" generally means "add more training data" rather than "we require significant conceptual breakthroughs".

That is not generally true. For example, GANs work great on MNIST, pretty well on the flower dataset, and ok on bedrooms.

But the same techniques currently fail on ImageNet - which actually is a much larger dataset. "Add more training data" is not a magic solution that overcomes limitations of your model.

In particular if you look at the generative models these GANs map to, it makes sense that they can learn 2D shapes and texture patterns, but rendering a complex 3D scene with significant depth complexity and lighting interactions is an entirely different beast. That problem has been studied deeply in 3D computer graphics and the generative programs successful there are vastly more complex than current GANs.

Re: Text to Image Synthesis Using Thought Vectors

#23
post #18

Earlier quoted context omitted.

But the thing about machine learning is that once it works at all, "a long way to go" generally means "add more training data" rather than "we require significant conceptual breakthroughs".

That is not generally true. For example, GANs work great on MNIST, pretty well on the flower dataset, and ok on bedrooms. But the same techniques currently fail on ImageNet - which actually is a much larger dataset. "Add more training data" is not a magic solution that overcomes limitations of your model. In particular if you look at the generative models these GANs map to, it makes sense that they can learn 2D shape…

seen this? https://arxiv.org/abs/1605.09304 awesome generated images, from imagenet
Post reply on HN