Image GPT • Hacker News Board

Re: Image GPT

#61

Post by gwern » Thu, Jun 18, 2020, 2:05 AM UTC

boulos wrote: post #46

jszymborski wrote: post #17

Apparently this takes "2500-V100 days" which is an insane amount of resources for images of this resolution. For context, this is equivalent to 100 $10,000 GPUs running for 25 days, 24/7. https://twitter.com/jm_alexia/status/1273349716915470340?s=2...

Disclosure: I work on Google Cloud (and know the OpenAI folks). Fwiw, I think 2500 "V100-days" here is an extrapolation of ~1 2048-TPUv3 day. So approximately $8/hr * 24 hours * 2048 => $400k, if you know, you could make use of that TPU Pod for much of the rest of the days of the year :). But yes, this is still the realm of "Do you have millions of dollars of ML infrastructure" (if you want it quickly). I'm kind of h…

We've moved up in the world! These days we'd train on one of our TFRC TPU pods. Colab is a lot of trouble and better avoided...

(We actually can create a TPU-2048 pod. For a short while, anyway, before it preempts. But we wouldn't need to if we used lucidrain's efficient attention implementations I mention in my other comment.)

Re: Image GPT

#62

Post by gwern » Thu, Jun 18, 2020, 2:06 AM UTC

gwern wrote: post #43

tux3 wrote: post #11

I'd be surprised if this architecture scales to larger resolutions, but any move towards "general learning" is really the interesting next step to me, not scaling up an inefficient architecture. Can they train the same GPT model on both text and images tasks at once, and would either task benefit at all from training on the other task? Even GPT-3 seems to have trouble with world-modeling, it writes convincing text th…

Of course it's not going to scale much past this, it's quadratic and already hitting painful compute levels. However, if you were starting this research today, you'd use any of half-a-dozen different self-attention variants which are roughly linear, including OA's own Sparse Transformers (which they did use to generate images, just on a far smaller scale which wouldn't be adequate to show competitive performance with…

* lucidrains, not Aydao

Re: Image GPT

#63

Post by gwern » Thu, Jun 18, 2020, 2:07 AM UTC

aabhay wrote: post #30

In the design field, there’s an adage — constraints inspire creativity. This work seems so unconstrained in its use of computation, that is almost screams to me that they must be going about it the wrong way.

We don't want 'creativity'. (Elegance is, as Boltzmann said, for tailors.) We want power.

Re: Image GPT

#64

Post by markchen90 » Thu, Jun 18, 2020, 2:11 AM UTC

aquajet wrote: post #55

markchen90 wrote: post #44

Earlier quoted context omitted.

Author here. I ran some early experiments a while ago, and it looked like adversarial examples for convnet classifiers didn't transfer to transformer classifiers and vice versa. Definitely worth looking more into!

Could you elaborate a bit more? What were the differences between transformer adversarial examples and cnn adversarial examples?

I didn't notice any obvious visual differences, but I'm also not an expert on adversarial examples. The transformer models were similarly susceptible to attacks, but while adversarial examples transferred well within a model class (~40%), they did not across model classes (~5%). These are rough numbers from memory, don't hold me accountable!

Re: Image GPT

#65

Post by superasn » Thu, Jun 18, 2020, 2:25 AM UTC

CShorten wrote: post #59

I made a video explaining this paper if interested! https://youtu.be/7rFLnQdl22c

What an excellent video. I know very little about this field but I was able to make sense of the things you've explained. I need to learn some more fundamentals I guess, but I'll surely be revisiting your video after that.

Re: Image GPT

#66

Post by samgriesemer » Thu, Jun 18, 2020, 2:37 AM UTC

aabhay wrote: post #30

In the design field, there’s an adage — constraints inspire creativity. This work seems so unconstrained in its use of computation, that is almost screams to me that they must be going about it the wrong way.

You might take a look at The Bitter Lesson [1], it's referenced by the article and linked around on this thread.

> One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The two methods that seem to scale arbitrarily in this way are search and learning.

> The second general point to be learned from the bitter lesson is that the actual contents of minds are tremendously, irredeemably complex; we should stop trying to find simple ways to think about the contents of minds, such as simple ways to think about space, objects, multiple agents, or symmetries.

[1]: http://www.incompleteideas.net/IncIdeas/BitterLesson.html

Re: Image GPT

#67

Post by boulos » Thu, Jun 18, 2020, 2:50 AM UTC

gwern wrote: post #61

boulos wrote: post #46

Earlier quoted context omitted.

Disclosure: I work on Google Cloud (and know the OpenAI folks). Fwiw, I think 2500 "V100-days" here is an extrapolation of ~1 2048-TPUv3 day. So approximately $8/hr * 24 hours * 2048 => $400k, if you know, you could make use of that TPU Pod for much of the rest of the days of the year :). But yes, this is still the realm of "Do you have millions of dollars of ML infrastructure" (if you want it quickly). I'm kind of h…

We've moved up in the world! These days we'd train on one of our TFRC TPU pods. Colab is a lot of trouble and better avoided... (We actually can create a TPU-2048 pod. For a short while, anyway, before it preempts. But we wouldn't need to if we used lucidrain's efficient attention implementations I mention in my other comment.)

Oh, I missed that! Fwiw, I’m a big proponent of 1/4-pod slices anyway :).

Re: Image GPT

#68

Post by boulos » Thu, Jun 18, 2020, 2:52 AM UTC

tinco wrote: post #47

sillysaurusx wrote: post #36

Earlier quoted context omitted.

I’ll have to do a writeup sometime explaining why these numbers aren’t as surprising as they seem. They’re also not as wasteful as they seem. Roughly, the boxes would be turned on anyway. Might as well put them to work. And yeah, it draws more power when in use (by a lot). But the data center probably isn’t paying a huge premium on top of what they would already pay for electricity. So all that’s left is feeling gene…

At some point it becomes linear though, there is a base cost to electricity. Google builds datacenters near coal power plants specifically to get (literally) dirt cheap electricity.

Just a correction here: we build datacenters near large sources of renewable / hydro plants (I consider hydro a borderline case). We buy renewable credits for everything and are even trying to timeshift load now to match renewable generation peaks.

tl;dr: not coal plants :).

Re: Image GPT

#69

Post by gowld » Thu, Jun 18, 2020, 3:16 AM UTC

gowld wrote: post #28

jszymborski wrote: post #17

Apparently this takes "2500-V100 days" which is an insane amount of resources for images of this resolution. For context, this is equivalent to 100 $10,000 GPUs running for 25 days, 24/7. https://twitter.com/jm_alexia/status/1273349716915470340?s=2...

Cost to train the model (which can be used for many images, and images can be upscaled using AI): $10000 * 2500days / 5yr = $15000 hardware cost 200W * 2500day * (0.10 USD / Whr) = $1.2M in electricity

$.1 for KWhr not When, so $1200 electricity.

Good use of units makes it easy to spot and fix the mistake.

Re: Image GPT

#70

Post by bjourne » Thu, Jun 18, 2020, 3:52 AM UTC

forgingahead wrote: post #60

boulos wrote: post #46

Earlier quoted context omitted.

Disclosure: I work on Google Cloud (and know the OpenAI folks). Fwiw, I think 2500 "V100-days" here is an extrapolation of ~1 2048-TPUv3 day. So approximately $8/hr * 24 hours * 2048 => $400k, if you know, you could make use of that TPU Pod for much of the rest of the days of the year :). But yes, this is still the realm of "Do you have millions of dollars of ML infrastructure" (if you want it quickly). I'm kind of h…

Is there a way to guarantee a specific GPU for Colab, and also have it run in the background without the runtime disconnecting? Even Colab Pro doesn't seem to guarantee a specific resource without the disconnects.

Nope. You need to have it open in a browser tab and to keep it from disconnecting you run this script:

    function ConnectButton(){document.querySelector("#top-toolbar > colab-connect-button").shadowRoot.querySelector("#connect").click(); console.log("Connect pushed"); } setInterval(ConnectButton,60000);

:) You still get booted every 6 hour or so.