Image GPT
openai.com
Image GPT
1–10 of 83 posts
Re: Image GPT
#2"As further proof, features from the model achieve state-of-the-art performance on a number of classification datasets and near state-of-the-art unsupervised accuracy on ImageNet."
Impressive stuff! This performs well even without domain-specific architecture choices.
Re: Image GPT
#3The model is open sourced on GitHub: https://github.com/openai/image-gpt
Oddly, it still uses TensorFlow like the original GPT-2 release despite OpenAI's declared switch to PyTorch, and it has dependency hell so it's not easy to create a wrapper tool for it.
Since it's still the GPT-2 architecture, it might be possible to port the weights to Huggingface Transformers (for the RGB generation), and then write a wrapper to extend it for the image rendering. (filed an issue here: https://github.com/huggingface/transformers/issues/5088 )
Re: Image GPT
#4The model is open sourced on GitHub: https://github.com/openai/image-gpt Oddly, it still uses TensorFlow like the original GPT-2 release despite OpenAI's declared switch to PyTorch, and it has dependency hell so it's not easy to create a wrapper tool for it. Since it's still the GPT-2 architecture, it might be possible to port the weights to Huggingface Transformers (for the RGB generation), and then write a wrapper…
It is quite amazing that this model is so concise and can be expressed with just a few hundred lines of code. (similar models as well, in general)
Looking at the samples I am amazed by some of the results.
Re: Image GPT
#5"As further proof, features from the model achieve state-of-the-art performance on a number of classification datasets and near state-of-the-art unsupervised accuracy on ImageNet." Impressive stuff! This performs well even without domain-specific architecture choices.
One more step for ML. It used to be that we needed hand designed image features. Now we can learn even the image priors (spatial locality and translation invariance) from data.
Transformers are basically learning relations between pairs of input tokens, moving the problem to a more abstract level than predicting directly on tokens. While CNNs excel at benefiting from those two forms of invariance, transformers have permutation invariance, they can predict on sets, graphs and non-euclidean spaces.
Re: Image GPT
#6The model is open sourced on GitHub: https://github.com/openai/image-gpt Oddly, it still uses TensorFlow like the original GPT-2 release despite OpenAI's declared switch to PyTorch, and it has dependency hell so it's not easy to create a wrapper tool for it. Since it's still the GPT-2 architecture, it might be possible to port the weights to Huggingface Transformers (for the RGB generation), and then write a wrapper…
> Oddly, it still uses TensorFlow like the original GPT-2 release despite OpenAI's declared switch to PyTorch
Note the footnote in the paper where evaluations was interrupted by the move to MS Azure. This is relatively old work, and since it's literally GPT-2 but images, it's no surprise they didn't bother to rewrite it in PyTorch. If their next big from-scratch thing (presumably the multimodal GPT-like one-model-to-rule-them-all research I've been looking forward to ever since the TR article) is TensorFlow, then I'll be surprised.
Re: Image GPT
#7"As further proof, features from the model achieve state-of-the-art performance on a number of classification datasets and near state-of-the-art unsupervised accuracy on ImageNet." Impressive stuff! This performs well even without domain-specific architecture choices.
One more step for ML. It used to be that we needed hand designed image features. Now we can learn even the image priors (spatial locality and translation invariance) from data. Transformers are basically learning relations between pairs of input tokens, moving the problem to a more abstract level than predicting directly on tokens. While CNNs excel at benefiting from those two forms of invariance, transformers have p…
> Now we can learn even the image priors (spatial locality and translation invariance) from data.
Right. The attention layers even learn attention patterns which look like convolution layer kernels! But better, presumably: https://arxiv.org/abs/1911.03584
Re: Image GPT
#8So how would this model be used for a classification task?
Re: Image GPT
#9This looks like a new iteration on the Pixel-RNN idea: https://arxiv.org/pdf/1601.06759.pdf
GPT2 definitely looks better.
Re: Image GPT
#10So how would this model be used for a classification task?
By extracting the features for the image (that is, the encoding, as in transfer learning in computer vision or natural language processing with, for instance, VGG or Bert, respectively) and feeding this to the classifier.