Worth noting that the new 345M model is still far from the full 1.5B model they were avoiding to post. The headline makes it seem like they finally decided to give the full model, but it's just a slightly larger demo model.
OpenAI releases larger GPT-2 model
21–30 of 87 posts
Re: OpenAI releases larger GPT-2 model
#22Oh wow, I was literally just making a toy Discord bot for GPT-2. Guess I'll update it with the bigger model. EDIT: Done! It takes about 3x longer than it did before to generate a response, so if you try it be very patient. Also, I made this in three hours so I wouldn't be surprised if it goes up in flames at some point. https://github.com/ScottPeterJohnson/gpt2-discord
How long does it take, and which hardware do you use?
Re: OpenAI releases larger GPT-2 model
#23> Due to our concerns about malicious applications of the technology, we are not releasing the trained model.
Has anyone said when they are intending to release the full model? There's likely a lot of positive applications of this technology as well.
Re: OpenAI releases larger GPT-2 model
#24Notably, the 345M model (1.5 GB on disk) is big enough that it's pushing the limits of conventional GPUs, and an alternative method of finetuning the model ( https://github.com/nshepperd/gpt-2/commit/47df6da611716b4826... ) has to be used to prevent the GPU from going OOM. I'm working on tools to streamline GPT-2 text generation: I'm currently porting the code above to gpt-2-simple ( https://github.com/minimaxir/gpt-…
Update: 345M model support successfully added to gpt-2-simple and new release pushed to PyPI, plus I've updated the corresponding Colaboratory Notebook accordingly (https://colab.research.google.com/drive/1VLG8e7YSEwypxU-noRN...)
At a glance it's hard to see if the 345M model is "better" at the moment (it's all qualitative), but I'll be doing more testing. Unfortunately, the 345M model might be slightly too resource intensive for the API/client use case I wanted to do, so I'll likely be sticking with the 117M model for now.
Re: OpenAI releases larger GPT-2 model
#25Notably, the 345M model (1.5 GB on disk) is big enough that it's pushing the limits of conventional GPUs, and an alternative method of finetuning the model ( https://github.com/nshepperd/gpt-2/commit/47df6da611716b4826... ) has to be used to prevent the GPU from going OOM. I'm working on tools to streamline GPT-2 text generation: I'm currently porting the code above to gpt-2-simple ( https://github.com/minimaxir/gpt-…
This thing really does scare me and fascinate me at the same time. There's so many nefarious things you could do with this... Generate fake news. spam google. etc. One valuable use could be to generate comedy and parody. You could also make it to sabotage others too. You could set it lose on nazi forums and have them argue with bots constantly.
I think people are over-worried about fake material. We've been able to say lies since we became human. We've been able to print lies since printing. We can already edit photos. People routinely make fake screenshots of messages to defame others. Newspapers sometimes print fake stories based on lies given by their sources or which they fail to vet. But provenance still exists to prove things, not how convincing it looks.
Re: OpenAI releases larger GPT-2 model
#26> Due to our concerns about malicious applications of the technology, we are not releasing the trained model. Has anyone said when they are intending to release the full model? There's likely a lot of positive applications of this technology as well.
They said within 6 months, however by that time there will likely be something better.
Re: OpenAI releases larger GPT-2 model
#27Earlier quoted context omitted.
This thing really does scare me and fascinate me at the same time. There's so many nefarious things you could do with this... Generate fake news. spam google. etc. One valuable use could be to generate comedy and parody. You could also make it to sabotage others too. You could set it lose on nazi forums and have them argue with bots constantly.
I think people are over-worried about fake material. We've been able to say lies since we became human. We've been able to print lies since printing. We can already edit photos. People routinely make fake screenshots of messages to defame others. Newspapers sometimes print fake stories based on lies given by their sources or which they fail to vet. But provenance still exists to prove things, not how convincing it lo…
If you wanted to run a propaganda department though, you needed to be a government or a large news organization. Now it will be within the reach of 4chan.
Re: OpenAI releases larger GPT-2 model
#28Notably, the 345M model (1.5 GB on disk) is big enough that it's pushing the limits of conventional GPUs, and an alternative method of finetuning the model ( https://github.com/nshepperd/gpt-2/commit/47df6da611716b4826... ) has to be used to prevent the GPU from going OOM. I'm working on tools to streamline GPT-2 text generation: I'm currently porting the code above to gpt-2-simple ( https://github.com/minimaxir/gpt-…
> Show HN: Blogia, a blogging framework for Ruby
Wow, this sounded so believable I had to check if this exists or not. It seems it doesn't exist, yet :).
Re: OpenAI releases larger GPT-2 model
#29Earlier quoted context omitted.
>(e.g. Hacker News titles from a retrained 117M model: https://github.com/minimaxir/hacker-news-gpt-2 ) Wow, thats great. “The Bullshit Bubble” “Fuck you, Bootstrap” “We should give up on America” - they’re practically comedy, yet very believable too.
"How to Make a T-Shirt in 10,000 hours" is my favorite.
I think its just edged by "Why I'm ditching ebike hoses for barometers" for me
Though "Hacker makes an infinitely scaleable cup of coffee" gives it a good run for its money.
Re: OpenAI releases larger GPT-2 model
#30Notably, the 345M model (1.5 GB on disk) is big enough that it's pushing the limits of conventional GPUs, and an alternative method of finetuning the model ( https://github.com/nshepperd/gpt-2/commit/47df6da611716b4826... ) has to be used to prevent the GPU from going OOM. I'm working on tools to streamline GPT-2 text generation: I'm currently porting the code above to gpt-2-simple ( https://github.com/minimaxir/gpt-…
I haven't heard of gradient checkpointing yet, thank you for the link! Do you know how it compares to gradient accumulation? The latter basically reduces the batch size, but takes the sum of multiple gradients before actually performing an update, thereby having the same effect as the original batch size.
The generated titles are great! You can put them into hncynic (https://github.com/leod/hncynic) to get closer to a fully generated HN experience.