GPT-2: 6-Month Follow-Up
openai.com
GPT-2: 6-Month Follow-Up
1–10 of 98 posts
Re: GPT-2: 6-Month Follow-Up
#2For finetuning GPT-2 on custom text, my gpt-2-simple package (https://github.com/minimaxir/gpt-2-simple) gets close to going OOM when finetuning the 345M model, even on a 16GB VRAM server GPU. Doubling the size of the model with the 774M model might cause it to not work at all, so I’ll need to test.
Of course, the default output from the model might be sufficient, although it’ll take twice as long to generate text compared to the 345M which is slow even on a GPU.
How exactly the large GPT-2 models are deployed is a mystery I really wish was open-sourced more.
Re: GPT-2: 6-Month Follow-Up
#3Are there any applications for the GPT-2 models beyond text synthesis? Inference, question-answering, NER detection/disambiguation, anything like this?
Re: GPT-2: 6-Month Follow-Up
#4For finetuning GPT-2 on custom text, my gpt-2-simple package ( https://github.com/minimaxir/gpt-2-simple ) gets close to going OOM when finetuning the 345M model, even on a 16GB VRAM server GPU. Doubling the size of the model with the 774M model might cause it to not work at all, so I’ll need to test. Of course, the default output from the model might be sufficient, although it’ll take twice as long to generate text…
I've already tried training with nshepperd's codebase. Sampling works, but even with the memory checkpointing and freezing the embedding and using SGD rather than Adam, it OOMs on a 1080ti's 11GB. Either additional tricks or CPU training are going to be required.
Re: GPT-2: 6-Month Follow-Up
#5Are there any applications for the GPT-2 models beyond text synthesis? Inference, question-answering, NER detection/disambiguation, anything like this?
All of those were part of the original benchmarks GPT-2 was evaluated on.
Re: GPT-2: 6-Month Follow-Up
#6Hmm, no mention of Megatron in their timeline? https://nv-adlr.github.io/MegatronLM
Re: GPT-2: 6-Month Follow-Up
#7
Are there any real use case for GPT-2? Does it solve any problem?
I've read almost all state of the art leaderboards of all Nlp tasks of paperswithcode.com and truth is except text generation, openAI has not one state of the art, they are not even visible in leaderboards.
OpenAI is maybe the AI research center with the biggest funding and comparatively to other well known (Microsoft, Facebook, Google or even zalando..) they are the ones with the least results.
From my observations most SOTAs come from chineses researchers by far, followed by deepmind.
BTW isn't that a sad truth that not even one of all major AI actors has a draft of an AGI architecture, something comparable to CYC or opencog. https://wiki.opencog.org/w/CogPrime_Overview
Two other observations I would like to share: Many important NLP tasks have almost nobody publicly working on them it seems, on paperswithcode.com or NLP-progress (from github) some tasks have only one or two papers... And many others have not evolved since 2016. Most of the time it seems trivial to beat the old state of the art, just use BERT or XLnet on a task where nobody applied it before and hop, free state of the art for you! Yet researchers don't seems to chase those low hanging, high returns fruits. Also researchers seems to work a lot in isolation, many new generic improvements like new optimizers (RAdam for example) and new activation functions (Swish) allow to beat most of older state of the art on almost all task just by using them. Yet researchers will take years before using them because of an absurd inertia. Also unlike an open source program, BERT and XLnet have very low response and activity on github despite major open issues...
Re: GPT-2: 6-Month Follow-Up
#8Are there any applications for the GPT-2 models beyond text synthesis? Inference, question-answering, NER detection/disambiguation, anything like this?
BERT and its descendants do better at all of this, and are the industry standard now
https://arxiv.org/abs/1810.04805
Re: GPT-2: 6-Month Follow-Up
#9For finetuning GPT-2 on custom text, my gpt-2-simple package ( https://github.com/minimaxir/gpt-2-simple ) gets close to going OOM when finetuning the 345M model, even on a 16GB VRAM server GPU. Doubling the size of the model with the 774M model might cause it to not work at all, so I’ll need to test. Of course, the default output from the model might be sufficient, although it’ll take twice as long to generate text…
I've already tried training with nshepperd's codebase. Sampling works, but even with the memory checkpointing and freezing the embedding and using SGD rather than Adam, it OOMs on a 1080ti's 11GB. Either additional tricks or CPU training are going to be required.
Are you using FP16?
Re: GPT-2: 6-Month Follow-Up
#10For finetuning GPT-2 on custom text, my gpt-2-simple package ( https://github.com/minimaxir/gpt-2-simple ) gets close to going OOM when finetuning the 345M model, even on a 16GB VRAM server GPU. Doubling the size of the model with the 774M model might cause it to not work at all, so I’ll need to test. Of course, the default output from the model might be sufficient, although it’ll take twice as long to generate text…
I've already tried training with nshepperd's codebase. Sampling works, but even with the memory checkpointing and freezing the embedding and using SGD rather than Adam, it OOMs on a 1080ti's 11GB. Either additional tricks or CPU training are going to be required.
Figured. I'll make changes to allow sampling from the default model more easily.