61–70 of 98 posts
Earlier quoted context omitted.
they already use internally a POS tagger and a dependency parser. Interesting. Where did you see that?
Well it was just a belief. I may be wrong. I asked them by curiosity https://github.com/openai/gpt-2/issues/168 So we will know.
Earlier quoted context omitted.
I'm the lead researcher on the Middlebury Institute project looking at fine-tuning the bigger models, and I got OOM on 745M and 1.5B originally. I had to get an Azure instance with 24GB VRAM to handle it (using nshepperd's codebase). It works, but takes a while (~500 epochs takes 12 hours on a 100k word training dataset).
Ouch! So 11GB is nowhere close to being enough, then. I wonder if even switching to FP16 will be adequate?
For finetuning GPT-2 on custom text, my gpt-2-simple package ( https://github.com/minimaxir/gpt-2-simple ) gets close to going OOM when finetuning the 345M model, even on a 16GB VRAM server GPU. Doubling the size of the model with the 774M model might cause it to not work at all, so I’ll need to test. Of course, the default output from the model might be sufficient, although it’ll take twice as long to generate text…
Possibly a stupid question, but does AMD lift such restrictions on models with its unified memory, by allowing the GPU to "page out" chunks of vram to system ram?
Are there any real use case for GPT-2? Does it solve any problem? I've read almost all state of the art leaderboards of all Nlp tasks of paperswithcode.com and truth is except text generation, openAI has not one state of the art, they are not even visible in leaderboards. OpenAI is maybe the AI research center with the biggest funding and comparatively to other well known (Microsoft, Facebook, Google or even zalando.…
If it was a high-return fruit somebody would be doing it. Not necessarily publishing papers about it or trying to beat useless artificial benchmarks on it.
The problem with this approach is that the users most likely to be malicious users of GPT2 are state actors. China, for example, already spends millions on an immense propaganda factory. Money is not a serious obstacle for a state. Given that other research entities are, by the sound of things, already far along with development of similar models it seems unlikely that China and the US don't already have functional models internally.
On the other hand, legitimate business and research is clearly hamstrung by withholding the full model. What we have is the maximum degree of inconvenience and the minimum degree of security. It feels almost perfectly analogous to ban on liquids in airports. The motivation for that ban was that existing security measures couldn't detect liquids, but simply announcing a ban was to be enforced didn't change the fact that liquids were undetectable. Instead millions of travelers were pointlessly inconvenienced at great cost.
Release the kraken already!
Are there any applications for the GPT-2 models beyond text synthesis? Inference, question-answering, NER detection/disambiguation, anything like this?
BERT and its descendants do better at all of this, and are the industry standard now https://arxiv.org/abs/1810.04805
I'm curious about the "fine-tuning based detection" mentioned in the report ("Fine-tunes a language model to 'detect itself'... over a range of available settings"). Does anyone know good articles/papers (or have an off-the-top tl;dr) to get a high-level grasp of "self-detection" for generative models?
The OpenAI approach to managing the release of the larger dataset strikes me as totally flawed and upside down. The biggest concern the team seem to have is that the fully trained GPT2 model will be used to spread propaganda and misinformation. They also imply that the biggest hurdle to training a similar model is money needed to pay for the training resources. The problem with this approach is that the users most li…
For finetuning GPT-2 on custom text, my gpt-2-simple package ( https://github.com/minimaxir/gpt-2-simple ) gets close to going OOM when finetuning the 345M model, even on a 16GB VRAM server GPU. Doubling the size of the model with the 774M model might cause it to not work at all, so I’ll need to test. Of course, the default output from the model might be sufficient, although it’ll take twice as long to generate text…