Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Image GPT

openai.com

Post reply on HN
Print view View original

51–60 of 83 posts

Re: Image GPT

#51

Post by dfischer »

slavik81 wrote: post #39
gowld wrote: post #28

Earlier quoted context omitted.

Cost to train the model (which can be used for many images, and images can be upscaled using AI): $10000 * 2500days / 5yr = $15000 hardware cost 200W * 2500day * (0.10 USD / Whr) = $1.2M in electricity

If electricity cost $0.1/Whr, a cup of tea would cost ~$10 just to run the kettle. You've overestimated the cost of electricity by roughly a thousand-fold. Residential electricity prices in California are more like 0.19 USD/kWh, which is 0.00019 USD/Wh.

I wonder what the cost is when taking second order effects into the equation. Leverage hides the true cost, burying the reality further and further from the obvious.

Re: Image GPT

#52

Post by markchen90 »

turdnagel wrote: post #50

The post states, "When we train GPT-2 on images unrolled into long sequences of pixels, which we call iGPT, we find that the model appears to understand 2-D image characteristics such as object appearance and category." This, to me (former philosophy student) feels like a very low bar for "understanding." Could the model _explain_ 2D image characteristics, or can it only generate them? I'm sure this debate will rage…

Author here. You're absolutely right that "understanding" is a fuzzy word. As you pointed out, part of reason we hold this belief is that the model can generate diverse samples and successfully complete out-of-distribution inputs. But the other part is that the model learns (without labels) useful features for classifying objects. Would be very interesting to test it on a broader set of datasets which measure other 2D image characteristics.

Re: Image GPT

#53

Post by xvedejas »

turdnagel wrote: post #50

The post states, "When we train GPT-2 on images unrolled into long sequences of pixels, which we call iGPT, we find that the model appears to understand 2-D image characteristics such as object appearance and category." This, to me (former philosophy student) feels like a very low bar for "understanding." Could the model _explain_ 2D image characteristics, or can it only generate them? I'm sure this debate will rage…

I work in computer vision; "understanding" here is meant to mean there is a statistical understanding. The model better represents the distribution of things that look like real images. I must say these examples are incredibly good at generating coherent scenes compared to any similar attempts I've seen. I do think it represents an advance in the that kind of understanding.

I think your definition of "understanding" is more the realm of AI, and why so many people in my field sigh when the term "AI" is used when we're really just talking about ML. But then again, you could absolutely in the current day train an ML model to look at images and then produce a natural language "explanation" of the image. It might not be able to make leaps in deductive logic, but it would be able to explain more than just a list of what is in the image. Is this "understanding"? Maybe that question is philosophy.

Re: Image GPT

#54

Post by inertiatic »

turdnagel wrote: post #50

The post states, "When we train GPT-2 on images unrolled into long sequences of pixels, which we call iGPT, we find that the model appears to understand 2-D image characteristics such as object appearance and category." This, to me (former philosophy student) feels like a very low bar for "understanding." Could the model _explain_ 2D image characteristics, or can it only generate them? I'm sure this debate will rage…

A convolutional net learns kernels that most people would say display some understanding over the domain of images as well. At points they might even serve as explanations. I think taking "understanding" to be "able to explain rigorously" is the highest bar possible here. A lot of people understand concepts that they are unable to explain. Animals understand a lot but lack a way of explaining things as well.

Re: Image GPT

#55

Post by aquajet »

markchen90 wrote: post #44
janhenr wrote: post #32

Earlier quoted context omitted.

Well, there is a whole seperate line of research concerning the topic of these input perturbations ranging from PGD to just Gaussian noise. This model does not claim to defend against any of those.

Author here. I ran some early experiments a while ago, and it looked like adversarial examples for convnet classifiers didn't transfer to transformer classifiers and vice versa. Definitely worth looking more into!

Could you elaborate a bit more? What were the differences between transformer adversarial examples and cnn adversarial examples?

Re: Image GPT

#56

Post by currymj »

AaronFriel wrote: post #18
minimaxir wrote: post #3

The model is open sourced on GitHub: https://github.com/openai/image-gpt Oddly, it still uses TensorFlow like the original GPT-2 release despite OpenAI's declared switch to PyTorch, and it has dependency hell so it's not easy to create a wrapper tool for it. Since it's still the GPT-2 architecture, it might be possible to port the weights to Huggingface Transformers (for the RGB generation), and then write a wrapper…

I'm still learning from deep learning papers and videos before dipping my toes in myself, is there a summary of why PyTorch vs TensorFlow? Does it matter for me?

just use PyTorch, it’s generally going to be the nicest.

TF2 is much nicer than TF1 but most of the changes were just to make it more like PyTorch.

Re: Image GPT

#57

Post by abledon »

i can imagine that in 15-25 years, consuming too much content esp. "AI" generated content will have guidelines around it. The possibilities with this stuff is just starting, it could 'overload' youtube and other hosting services once 'creators' begin to normalize its power (10 yrs from now thers an easy photoshp/adobe plugin to generate random video/image scenes with actors and generated voices/movement (animated rigs aka mixamo))

Re: Image GPT

#58

Post by sillysaurusx »

tinco wrote: post #47
sillysaurusx wrote: post #36

Earlier quoted context omitted.

I’ll have to do a writeup sometime explaining why these numbers aren’t as surprising as they seem. They’re also not as wasteful as they seem. Roughly, the boxes would be turned on anyway. Might as well put them to work. And yeah, it draws more power when in use (by a lot). But the data center probably isn’t paying a huge premium on top of what they would already pay for electricity. So all that’s left is feeling gene…

At some point it becomes linear though, there is a base cost to electricity. Google builds datacenters near coal power plants specifically to get (literally) dirt cheap electricity.

This is true. But researchers are divorced from those realities. The day-to-day researchers almost never have to think like "If I run this, will it cost a ton?" Those kinds of agreements are usually made on a company basis, i.e. there's no cost to just leave a model running and see what happens. (Which is a huge advantage, by the way.)

Re: Image GPT

#60

Post by forgingahead »

boulos wrote: post #46
jszymborski wrote: post #17

Apparently this takes "2500-V100 days" which is an insane amount of resources for images of this resolution. For context, this is equivalent to 100 $10,000 GPUs running for 25 days, 24/7. https://twitter.com/jm_alexia/status/1273349716915470340?s=2...

Disclosure: I work on Google Cloud (and know the OpenAI folks). Fwiw, I think 2500 "V100-days" here is an extrapolation of ~1 2048-TPUv3 day. So approximately $8/hr * 24 hours * 2048 => $400k, if you know, you could make use of that TPU Pod for much of the rest of the days of the year :). But yes, this is still the realm of "Do you have millions of dollars of ML infrastructure" (if you want it quickly). I'm kind of h…

Is there a way to guarantee a specific GPU for Colab, and also have it run in the background without the runtime disconnecting? Even Colab Pro doesn't seem to guarantee a specific resource without the disconnects.
Post reply on HN