21–30 of 83 posts
Earlier quoted context omitted.
No, this is a new iteration on the AIAYN idea.
What's this ?
It reflects the evolution of NLP models: (RNN) —> (RNN + attention) —> (attention) and the idea that you don’t need the recurrent component. Just look at all elements of the sequence at once, applying varying weights (attention).
The model is open sourced on GitHub: https://github.com/openai/image-gpt Oddly, it still uses TensorFlow like the original GPT-2 release despite OpenAI's declared switch to PyTorch, and it has dependency hell so it's not easy to create a wrapper tool for it. Since it's still the GPT-2 architecture, it might be possible to port the weights to Huggingface Transformers (for the RGB generation), and then write a wrapper…
I'm still learning from deep learning papers and videos before dipping my toes in myself, is there a summary of why PyTorch vs TensorFlow? Does it matter for me?
I noted the TensorFlow usage because the original GPT-2 release was TensorFlow 1.X, which led to issues when TensorFlow 2.0 was released soon after.
For model training, I strongly recommend using the higher-level Keras APIs for TensorFlow/pytorch-lightning API for PyTorch than the respective base tools.
I'd be surprised if this architecture scales to larger resolutions, but any move towards "general learning" is really the interesting next step to me, not scaling up an inefficient architecture. Can they train the same GPT model on both text and images tasks at once, and would either task benefit at all from training on the other task? Even GPT-3 seems to have trouble with world-modeling, it writes convincing text th…
> I'd be surprised if this architecture scales to larger resolutions Could a different NN be used for upscaling?
Far more interesting would be general training optimizations that could train GPT faster on any kind of task.
Apparently this takes "2500-V100 days" which is an insane amount of resources for images of this resolution. For context, this is equivalent to 100 $10,000 GPUs running for 25 days, 24/7. https://twitter.com/jm_alexia/status/1273349716915470340?s=2...
This is truly a brute force approach.
Apparently this takes "2500-V100 days" which is an insane amount of resources for images of this resolution. For context, this is equivalent to 100 $10,000 GPUs running for 25 days, 24/7. https://twitter.com/jm_alexia/status/1273349716915470340?s=2...
$10000 * 2500days / 5yr = $15000 hardware cost
200W * 2500day * (0.10 USD / Whr) = $1.2M in electricity
This may be my imagination, or maybe it’s just because the images are so small, but these seem to have fewer of the slightly disturbing artifacts you see in CNN generated images.
This work seems so unconstrained in its use of computation, that is almost screams to me that they must be going about it the wrong way.