OpenAI releases larger GPT-2 model
openai.com
OpenAI releases larger GPT-2 model
1–10 of 87 posts
Re: OpenAI releases larger GPT-2 model
#2Notably, the 345M model (1.5 GB on disk) is big enough that it's pushing the limits of conventional GPUs, and an alternative method of finetuning the model (https://github.com/nshepperd/gpt-2/commit/47df6da611716b4826...) has to be used to prevent the GPU from going OOM.
I'm working on tools to streamline GPT-2 text generation: I'm currently porting the code above to gpt-2-simple (https://github.com/minimaxir/gpt-2-simple) to allow easy finetuning/generation, and am also working on a way to quickly build an API/client for easily deploying GPT-2 to production and generating text at scale and cost effectively. Even the 117M model, managing CPU and RAM performance is tricky.
But given the incredible results of just the 117M model (e.g. Hacker News titles from a retrained 117M model: https://github.com/minimaxir/hacker-news-gpt-2), I'm eager to put the 345M model through its paces.
Re: OpenAI releases larger GPT-2 model
#3Notably, the 345M model (1.5 GB on disk) is big enough that it's pushing the limits of conventional GPUs, and an alternative method of finetuning the model ( https://github.com/nshepperd/gpt-2/commit/47df6da611716b4826... ) has to be used to prevent the GPU from going OOM. I'm working on tools to streamline GPT-2 text generation: I'm currently porting the code above to gpt-2-simple ( https://github.com/minimaxir/gpt-…
Can you elaborate on the expected hardware requirements? (Both for fine tuning and inference)
Re: OpenAI releases larger GPT-2 model
#4Notably, the 345M model (1.5 GB on disk) is big enough that it's pushing the limits of conventional GPUs, and an alternative method of finetuning the model ( https://github.com/nshepperd/gpt-2/commit/47df6da611716b4826... ) has to be used to prevent the GPU from going OOM. I'm working on tools to streamline GPT-2 text generation: I'm currently porting the code above to gpt-2-simple ( https://github.com/minimaxir/gpt-…
It seems to train fine on Colab without that flag, so that's a decent fallback for anyone running into memory issues on a local machine.
Re: OpenAI releases larger GPT-2 model
#5Notably, the 345M model (1.5 GB on disk) is big enough that it's pushing the limits of conventional GPUs, and an alternative method of finetuning the model ( https://github.com/nshepperd/gpt-2/commit/47df6da611716b4826... ) has to be used to prevent the GPU from going OOM. I'm working on tools to streamline GPT-2 text generation: I'm currently porting the code above to gpt-2-simple ( https://github.com/minimaxir/gpt-…
>(e.g. Hacker News titles from a retrained 117M model: https://github.com/minimaxir/hacker-news-gpt-2)
Wow, thats great. “The Bullshit Bubble” “Fuck you, Bootstrap” “We should give up on America” - they’re practically comedy, yet very believable too.
Re: OpenAI releases larger GPT-2 model
#6Notably, the 345M model (1.5 GB on disk) is big enough that it's pushing the limits of conventional GPUs, and an alternative method of finetuning the model ( https://github.com/nshepperd/gpt-2/commit/47df6da611716b4826... ) has to be used to prevent the GPU from going OOM. I'm working on tools to streamline GPT-2 text generation: I'm currently porting the code above to gpt-2-simple ( https://github.com/minimaxir/gpt-…
It seems to train fine on Colab without that flag, so that's a decent fallback for anyone running into memory issues on a local machine.
If you did a fresh pull of that fork using a Colab notebook that does so, it will set the flag automatically if you specify the 345M model (I will follow that behavior in my own package).
If it didn't set the flag, it gets messy: https://twitter.com/minimaxir/status/1124742105421631488
Re: OpenAI releases larger GPT-2 model
#7Notably, the 345M model (1.5 GB on disk) is big enough that it's pushing the limits of conventional GPUs, and an alternative method of finetuning the model ( https://github.com/nshepperd/gpt-2/commit/47df6da611716b4826... ) has to be used to prevent the GPU from going OOM. I'm working on tools to streamline GPT-2 text generation: I'm currently porting the code above to gpt-2-simple ( https://github.com/minimaxir/gpt-…
One example from your dataset that gave me a laugh:
Hiring technical debt (or "unsortable overtime")
How do you hire technical debt??
Edit: another one-
“In 2009, Africa power creation was switched on for the Google Earth Darth Vader Imperial Warplane Propaganda”
I guess Google is diversifying :P
Re: OpenAI releases larger GPT-2 model
#8[deleted]
Re: OpenAI releases larger GPT-2 model
#9If you guys want some background check out this blog post:
https://openai.com/blog/better-language-models/
It's fascinating to think an AI wrote that story...
Re: OpenAI releases larger GPT-2 model
#10Notably, the 345M model (1.5 GB on disk) is big enough that it's pushing the limits of conventional GPUs, and an alternative method of finetuning the model ( https://github.com/nshepperd/gpt-2/commit/47df6da611716b4826... ) has to be used to prevent the GPU from going OOM. I'm working on tools to streamline GPT-2 text generation: I'm currently porting the code above to gpt-2-simple ( https://github.com/minimaxir/gpt-…
This thing really does scare me and fascinate me at the same time.
There's so many nefarious things you could do with this...
Generate fake news. spam google. etc.
One valuable use could be to generate comedy and parody.
You could also make it to sabotage others too. You could set it lose on nazi forums and have them argue with bots constantly.