For finetuning GPT-2 on custom text, my gpt-2-simple package ( https://github.com/minimaxir/gpt-2-simple ) gets close to going OOM when finetuning the 345M model, even on a 16GB VRAM server GPU. Doubling the size of the model with the 774M model might cause it to not work at all, so I’ll need to test. Of course, the default output from the model might be sufficient, although it’ll take twice as long to generate text…
Possibly a stupid question, but does AMD lift such restrictions on models with its unified memory, by allowing the GPU to "page out" chunks of vram to system ram?
Are there any real use case for GPT-2? Does it solve any problem? I've read almost all state of the art leaderboards of all Nlp tasks of paperswithcode.com and truth is except text generation, openAI has not one state of the art, they are not even visible in leaderboards. OpenAI is maybe the AI research center with the biggest funding and comparatively to other well known (Microsoft, Facebook, Google or even zalando.…
The article mentions TabNine, which has made it to the HN frontpage before.
For finetuning GPT-2 on custom text, my gpt-2-simple package ( https://github.com/minimaxir/gpt-2-simple ) gets close to going OOM when finetuning the 345M model, even on a 16GB VRAM server GPU. Doubling the size of the model with the 774M model might cause it to not work at all, so I’ll need to test. Of course, the default output from the model might be sufficient, although it’ll take twice as long to generate text…
Possibly a stupid question, but does AMD lift such restrictions on models with its unified memory, by allowing the GPU to "page out" chunks of vram to system ram?
No idea how modeling works on AMD. (most discussions are about NVidia/CUDA)
Why the ad hominem? I am pointing a problem of allocation of ressources on the AI research field. It's not to me to fixe that, but yes I am actively working on a logical fallacies detector which is the first of human history and works for the 256 possible forms of syllogisms, I'm expanding it to other logical forms such as modus ponens/tollens.
It's not to me to fixe that There's nothing to fix. People work on what they want to work on. Things that seem important to you are not important to me, and the opposite. I'm OK with that.
"People work on what they want to work" ideally yes, but ultimately they work on something that please them AND that give them a decent salary.
Funding should not go to fun (but useless in the real world) Nlp tasks.
"Things that seem important to you are not important to me, and the opposite." and here's go relativism or the abandon of thought...
It's indeed difficult to quantify cardinally the utility of an NLP task against an other, but we can agree on an ordinality (order of magnitude)
E.g do you understand that POS tagging or dependency/constictuency parsing are angular tasks needed by much of the others. Thus making them the most important NLP tasks as they enable other Nlp tasks and are the most used in practice?
You think that what exactly is more important?
Are you talking about text generation? Why is that important?
Something important enable to solve important problems in the real world.
How text generation solve any real world problem is beyond my knowledge.
But if you rationally think that it's more important that angular Nlp tasks, you can probably explain why and give an example or two?
Yes, an AGI will need to emit text just as humans do, indeed.
But before that she needs to understund the natural language before emitting it.
GPT-2 maybe capture an aesthetic of the initial input pretty well but it does not generate meaningful sentences or only by accident, so no GPT-2 does not advance the quest to create an intelligent agent mastering natural language.
I was able to take all of Donald Trumps tweets and using GPT2 to make a program that would mimic his tweets. I found that it might be very effective. I have the test at https://docs.google.com/forms/d/1p7tlobl5y5plBCu_enK4KawR7B8... I got the information from trumptwitterarchive.com I also explored creating a system that could recognize fake tweets from real ones and I believe I got 94% accuracy. It was a Bayes class…
"I also explored creating a system that could recognize fake tweets from real ones and I believe I got 94% accuracy. It was a Bayes classifier but I think I have to double check my work."
Is it open source?
This interest me a lot!
Are there any applications for the GPT-2 models beyond text synthesis? Inference, question-answering, NER detection/disambiguation, anything like this?
BERT and its descendants do better at all of this, and are the industry standard now https://arxiv.org/abs/1810.04805
Are there any real use case for GPT-2? Does it solve any problem? I've read almost all state of the art leaderboards of all Nlp tasks of paperswithcode.com and truth is except text generation, openAI has not one state of the art, they are not even visible in leaderboards. OpenAI is maybe the AI research center with the biggest funding and comparatively to other well known (Microsoft, Facebook, Google or even zalando.…
Many important NLP tasks have almost nobody publicly working on them Well, then perhaps you should go work on them, instead of ranting here.
Peer review is so shit at major AI conferences that his paper was most likely rejected for nonsensical reasons
Hopefully someone will make a working demo of it, like Adam King did for 345M. People should be able to experiment with this stuff without relying on the hype of press releases: https://medium.com/@VictorBanev/interrogating-gpt-2-345m-aaf... Not sure why open AI doesn't do this themselves. That fully aligns with their stated mission.
Are there any applications for the GPT-2 models beyond text synthesis? Inference, question-answering, NER detection/disambiguation, anything like this?
I really doubt it as they are not on any state of the art leaderboard from both NLP-progress and paperswithcode.com