Image GPT • Hacker News Board

Re: Image GPT

#81

Post by aabhay » Fri, Jun 19, 2020, 2:44 PM UTC

gwern wrote: post #63

aabhay wrote: post #30

In the design field, there’s an adage — constraints inspire creativity. This work seems so unconstrained in its use of computation, that is almost screams to me that they must be going about it the wrong way.

We don't want 'creativity'. (Elegance is, as Boltzmann said, for tailors.) We want power .

There’s a difference between creativity and elegance! Elegance is more about the beauty of the solution rather than the value of the outcome. The power of a blunt instrument is impotent at best, destructive at worst.

Re: Image GPT

#82

Post by visarga » Fri, Jun 19, 2020, 8:14 PM UTC

tux3 wrote: post #11

I'd be surprised if this architecture scales to larger resolutions, but any move towards "general learning" is really the interesting next step to me, not scaling up an inefficient architecture. Can they train the same GPT model on both text and images tasks at once, and would either task benefit at all from training on the other task? Even GPT-3 seems to have trouble with world-modeling, it writes convincing text th…

It's been done. For example it is possible to take an image through a CNN, generate box proposals and make a list of embeddings extracted from the boxes. That would be a bag of visual tokens. Then continue the sequence with text tokens and train it to solve a reasoning task.

Another example: Unicoder-VL https://arxiv.org/pdf/1908.06066.pdf

Re: Image GPT

#83

Post by visarga » Fri, Jun 19, 2020, 8:23 PM UTC

gauthamzz wrote: post #12

What are some real world use cases for something like this?

We now understand better the capabilities of the transformer, which is the hottest thing in AI in the last 2 years. The transformer apparently can learn images even if nobody tells it about the properties of space. CNN's on the other part rely heavily on these properties (spatial locality and translation invariance).