This looks like a new iteration on the Pixel-RNN idea: https://arxiv.org/pdf/1601.06759.pdf GPT2 definitely looks better.
No, this is a new iteration on the AIAYN idea.
Also an instance of The Bitter Lesson
Image GPT
31–40 of 83 posts
Re: Image GPT
#32As someone not up-to-date with literature, are one-pixel/few-pixels/small-delta attack issues resolved yet?
Well, there is a whole seperate line of research concerning the topic of these input perturbations ranging from PGD to just Gaussian noise. This model does not claim to defend against any of those.
Re: Image GPT
#33There's some definite overfitting apparent in the completion of the bottom half of the cat (blue bkg) photo. Every completion has a funny index card covering the bottom including the "original", while autocomplete should be more robust than recreating the original photo.
Re: Image GPT
#34The model is open sourced on GitHub: https://github.com/openai/image-gpt Oddly, it still uses TensorFlow like the original GPT-2 release despite OpenAI's declared switch to PyTorch, and it has dependency hell so it's not easy to create a wrapper tool for it. Since it's still the GPT-2 architecture, it might be possible to port the weights to Huggingface Transformers (for the RGB generation), and then write a wrapper…
> Oddly, it still uses TensorFlow like the original GPT-2 release despite OpenAI's declared switch to PyTorch Note the footnote in the paper where evaluations was interrupted by the move to MS Azure. This is relatively old work, and since it's literally GPT-2 but images, it's no surprise they didn't bother to rewrite it in PyTorch. If their next big from-scratch thing (presumably the multimodal GPT-like one-model-to-…
Unfortunately, there probably won’t be a next big-from-scratch thing unless it ties into their API somehow. It feels like we’ve seen the last of OpenAI open sourcing anything but toy models.
Of course, I would be delighted to be mistaken.
Re: Image GPT
#35Apparently this takes "2500-V100 days" which is an insane amount of resources for images of this resolution. For context, this is equivalent to 100 $10,000 GPUs running for 25 days, 24/7. https://twitter.com/jm_alexia/status/1273349716915470340?s=2...
Cost to train the model (which can be used for many images, and images can be upscaled using AI): $10000 * 2500days / 5yr = $15000 hardware cost 200W * 2500day * (0.10 USD / Whr) = $1.2M in electricity
Electricity rates are kWh, so I think it would be 120k for electricity, not 1.2 mil.
Re: Image GPT
#36Apparently this takes "2500-V100 days" which is an insane amount of resources for images of this resolution. For context, this is equivalent to 100 $10,000 GPUs running for 25 days, 24/7. https://twitter.com/jm_alexia/status/1273349716915470340?s=2...
Cost to train the model (which can be used for many images, and images can be upscaled using AI): $10000 * 2500days / 5yr = $15000 hardware cost 200W * 2500day * (0.10 USD / Whr) = $1.2M in electricity
I’ll have to do a writeup sometime explaining why these numbers aren’t as surprising as they seem. They’re also not as wasteful as they seem.
Roughly, the boxes would be turned on anyway. Might as well put them to work.
And yeah, it draws more power when in use (by a lot). But the data center probably isn’t paying a huge premium on top of what they would already pay for electricity.
So all that’s left is feeling generally queasy about using so much electricity, rather than a practical feeling of “this is bad because it costs a lot.”
Economies of scale have some dramatic effects at the high ends: I doubt the datacenter’s costs are anywhere near linear for the amount of Whr they’re consuming. And even if they are, it’s still a rounding error considering the profitability of GoogSoftBook.
Re: Image GPT
#37There's some definite overfitting apparent in the completion of the bottom half of the cat (blue bkg) photo. Every completion has a funny index card covering the bottom including the "original", while autocomplete should be more robust than recreating the original photo.
I think that the edge of the index card is included in the top half of the image.
Re: Image GPT
#38Earlier quoted context omitted.
I'm still learning from deep learning papers and videos before dipping my toes in myself, is there a summary of why PyTorch vs TensorFlow? Does it matter for me?
Nowadays , there isn't a huge practical difference in terms of performance/tooling aside from edge cases and deployment options. It mostly depends on your syntax preference. (although there are flame wars from both sides) I noted the TensorFlow usage because the original GPT-2 release was TensorFlow 1.X, which led to issues when TensorFlow 2.0 was released soon after. For model training, I strongly recommend using th…
There is a huge difference in developer experience still. When TF fails it typically offers little to no clue as to what you can do to rectify the situation. PyTorch is a lot more helpful most of the time. PyTorch is worth choosing on this basis alone, unless you intend to deploy to mobile hardware, where TFLite is the only real viable option.
Re: Image GPT
#39Apparently this takes "2500-V100 days" which is an insane amount of resources for images of this resolution. For context, this is equivalent to 100 $10,000 GPUs running for 25 days, 24/7. https://twitter.com/jm_alexia/status/1273349716915470340?s=2...
Cost to train the model (which can be used for many images, and images can be upscaled using AI): $10000 * 2500days / 5yr = $15000 hardware cost 200W * 2500day * (0.10 USD / Whr) = $1.2M in electricity
If electricity cost $0.1/Whr, a cup of tea would cost ~$10 just to run the kettle. You've overestimated the cost of electricity by roughly a thousand-fold.
Residential electricity prices in California are more like 0.19 USD/kWh, which is 0.00019 USD/Wh.
Re: Image GPT
#40Earlier quoted context omitted.
Cost to train the model (which can be used for many images, and images can be upscaled using AI): $10000 * 2500days / 5yr = $15000 hardware cost 200W * 2500day * (0.10 USD / Whr) = $1.2M in electricity
If electricity cost $0.1/Whr, a cup of tea would cost ~$10 just to run the kettle. You've overestimated the cost of electricity by roughly a thousand-fold. Residential electricity prices in California are more like 0.19 USD/kWh, which is 0.00019 USD/Wh.
Found the Brit.