Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Image GPT

openai.com

Post reply on HN
Print view View original

41–50 of 83 posts

Re: Image GPT

#41

Post by unwoundmouse »

tux3 wrote: post #11

I'd be surprised if this architecture scales to larger resolutions, but any move towards "general learning" is really the interesting next step to me, not scaling up an inefficient architecture. Can they train the same GPT model on both text and images tasks at once, and would either task benefit at all from training on the other task? Even GPT-3 seems to have trouble with world-modeling, it writes convincing text th…

This is the question I want answered. I feel that in the human brain, many words evoke imagery. For instance, when I hear the word bleak, I think of a grey setting

Re: Image GPT

#42

Post by [deleted] »

AaronFriel wrote: post #18
minimaxir wrote: post #3

The model is open sourced on GitHub: https://github.com/openai/image-gpt Oddly, it still uses TensorFlow like the original GPT-2 release despite OpenAI's declared switch to PyTorch, and it has dependency hell so it's not easy to create a wrapper tool for it. Since it's still the GPT-2 architecture, it might be possible to port the weights to Huggingface Transformers (for the RGB generation), and then write a wrapper…

I'm still learning from deep learning papers and videos before dipping my toes in myself, is there a summary of why PyTorch vs TensorFlow? Does it matter for me?

[deleted]

Re: Image GPT

#43

Post by gwern »

tux3 wrote: post #11

I'd be surprised if this architecture scales to larger resolutions, but any move towards "general learning" is really the interesting next step to me, not scaling up an inefficient architecture. Can they train the same GPT model on both text and images tasks at once, and would either task benefit at all from training on the other task? Even GPT-3 seems to have trouble with world-modeling, it writes convincing text th…

Of course it's not going to scale much past this, it's quadratic and already hitting painful compute levels.

However, if you were starting this research today, you'd use any of half-a-dozen different self-attention variants which are roughly linear, including OA's own Sparse Transformers (which they did use to generate images, just on a far smaller scale which wouldn't be adequate to show competitive performance with SimCLR etc). With those, it's perfectly possible to do self-attention over whole images at 256px or higher.

(As a matter of fact, Aydao has been working on a StyleGAN which just uses self-attention every layer instead of convolutions, using one of the new attentions; you can see some generated image samples from Flowers here: https://github.com/tensorfork/tensorfork/issues/31 GAN loss, not autoregressive pixel likelihood, but it makes the point.)

Re: Image GPT

#44

Post by markchen90 »

janhenr wrote: post #32
fizixer wrote: post #24

As someone not up-to-date with literature, are one-pixel/few-pixels/small-delta attack issues resolved yet?

Well, there is a whole seperate line of research concerning the topic of these input perturbations ranging from PGD to just Gaussian noise. This model does not claim to defend against any of those.

Author here. I ran some early experiments a while ago, and it looked like adversarial examples for convnet classifiers didn't transfer to transformer classifiers and vice versa. Definitely worth looking more into!

Re: Image GPT

#45

Post by fpgaminer »

In case anyone looking through the linked article is also wondering why the images look odd, it's because (buried in the middle):

> motivated by early color display palettes, we create our own 9-bit color palette to represent pixels. Using this palette yields an input sequence length 3 times shorter than the standard (R, G, B) palette, while still encoding color faithfully.

Makes total sense now. In fact, those images remind me so much of how photos looked on the early internet, since many were palletized.

Re: Image GPT

#46

Post by boulos »

jszymborski wrote: post #17

Apparently this takes "2500-V100 days" which is an insane amount of resources for images of this resolution. For context, this is equivalent to 100 $10,000 GPUs running for 25 days, 24/7. https://twitter.com/jm_alexia/status/1273349716915470340?s=2...

Disclosure: I work on Google Cloud (and know the OpenAI folks).

Fwiw, I think 2500 "V100-days" here is an extrapolation of ~1 2048-TPUv3 day. So approximately $8/hr * 24 hours * 2048 => $400k, if you know, you could make use of that TPU Pod for much of the rest of the days of the year :).

But yes, this is still the realm of "Do you have millions of dollars of ML infrastructure" (if you want it quickly). I'm kind of hoping gwern et al. will use Colab to slowly train a variant of this for free :).

Re: Image GPT

#47

Post by tinco »

sillysaurusx wrote: post #36
gowld wrote: post #28

Earlier quoted context omitted.

Cost to train the model (which can be used for many images, and images can be upscaled using AI): $10000 * 2500days / 5yr = $15000 hardware cost 200W * 2500day * (0.10 USD / Whr) = $1.2M in electricity

I’ll have to do a writeup sometime explaining why these numbers aren’t as surprising as they seem. They’re also not as wasteful as they seem. Roughly, the boxes would be turned on anyway. Might as well put them to work. And yeah, it draws more power when in use (by a lot). But the data center probably isn’t paying a huge premium on top of what they would already pay for electricity. So all that’s left is feeling gene…

At some point it becomes linear though, there is a base cost to electricity. Google builds datacenters near coal power plants specifically to get (literally) dirt cheap electricity.

Re: Image GPT

#48

Post by Scaevolus »

indiandennis wrote: post #35
gowld wrote: post #28

Earlier quoted context omitted.

Cost to train the model (which can be used for many images, and images can be upscaled using AI): $10000 * 2500days / 5yr = $15000 hardware cost 200W * 2500day * (0.10 USD / Whr) = $1.2M in electricity

Electricity rates are kWh, so I think it would be 120k for electricity, not 1.2 mil.

$1200, off by a factor of 1000.

Re: Image GPT

#50

Post by turdnagel »

The post states, "When we train GPT-2 on images unrolled into long sequences of pixels, which we call iGPT, we find that the model appears to understand 2-D image characteristics such as object appearance and category." This, to me (former philosophy student) feels like a very low bar for "understanding." Could the model _explain_ 2D image characteristics, or can it only generate them? I'm sure this debate will rage on for a while, but when it comes to intelligence, I believe we ought to be more rigorous with our use of the word.
Post reply on HN