Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-2: 6-Month Follow-Up

openai.com

Post reply on HN
Print view View original

51–60 of 98 posts

Re: GPT-2: 6-Month Follow-Up

#51

Post by p1esk »

The_rationalist wrote: post #45
p1esk wrote: post #43

Earlier quoted context omitted.

do you understand that POS tagging or dependency/constictuency parsing are angular tasks needed by much of the others. I'm not sure. I rarely have to do that explicitly in my head. Perhaps a model should learn to infer/guess them implicitly, from context, just like I do. what exactly is more important? In my opinion, having a world model (for common sense) and situational awareness (e.g. through sensor fusion, or fro…

I'm not sure. I rarely have to do that explicitly in my head. Well I can't prove it but I strongly believe that our brains use part of speech too, unconsciously. Perhaps a model should learn to infer/guess them implicitly, from data. That's exactly what deep learning POS tagger do, they are far better than hard coded algorithms. SOTA has 97.96% of accuracy. In my opinion, having a world model (for common sense) and s…

they already use internally a POS tagger and a dependency parser.

Interesting. Where did you see that?

Re: GPT-2: 6-Month Follow-Up

#52

Post by newhaus1994 »

gwern wrote: post #4
minimaxir wrote: post #2

For finetuning GPT-2 on custom text, my gpt-2-simple package ( https://github.com/minimaxir/gpt-2-simple ) gets close to going OOM when finetuning the 345M model, even on a 16GB VRAM server GPU. Doubling the size of the model with the 774M model might cause it to not work at all, so I’ll need to test. Of course, the default output from the model might be sufficient, although it’ll take twice as long to generate text…

I've already tried training with nshepperd's codebase. Sampling works, but even with the memory checkpointing and freezing the embedding and using SGD rather than Adam, it OOMs on a 1080ti's 11GB. Either additional tricks or CPU training are going to be required.

I'm the lead researcher on the Middlebury Institute project looking at fine-tuning the bigger models, and I got OOM on 745M and 1.5B originally. I had to get an Azure instance with 24GB VRAM to handle it (using nshepperd's codebase). It works, but takes a while (~500 epochs takes 12 hours on a 100k word training dataset).

Re: GPT-2: 6-Month Follow-Up

#53

Post by newhaus1994 »

gambler wrote: post #28

"Cornell University is studying human susceptibility to digital disinformation generated by language models." "The Middlebury Institute of International Studies Center on Terrorism, Extremism, and Counterterrorism (CTEC) is exploring how GPT-2 could be misused by terrorists and extremists online." "The University of Oregon is developing a series of “bias probes” to analyze bias within GPT-2." But apparently no univer…

We (the Middlebury Institute's CTEC) are an extremism and terrorism research lab, and so we're tracking the ways that tech is used by terrorists and extremists.

For a lot of nonstate orgs with sophisticated propaganda arms, an ideologically cohesive text generation capability would be a huge advantage in scaling up info ops. We are looking to measure whether or not GPT-2 or other neural text generators are useful for this, or if that risk is, as you say, nonsense.

Re: GPT-2: 6-Month Follow-Up

#54

Post by postalrat »

comradesmith wrote: post #47
gambler wrote: post #28

"Cornell University is studying human susceptibility to digital disinformation generated by language models." "The Middlebury Institute of International Studies Center on Terrorism, Extremism, and Counterterrorism (CTEC) is exploring how GPT-2 could be misused by terrorists and extremists online." "The University of Oregon is developing a series of “bias probes” to analyze bias within GPT-2." But apparently no univer…

http://talktotransformer.com Seems accessible to me

That isn't running the full model.

Re: GPT-2: 6-Month Follow-Up

#55

Post by The_rationalist »

p1esk wrote: post #51
The_rationalist wrote: post #45

Earlier quoted context omitted.

I'm not sure. I rarely have to do that explicitly in my head. Well I can't prove it but I strongly believe that our brains use part of speech too, unconsciously. Perhaps a model should learn to infer/guess them implicitly, from data. That's exactly what deep learning POS tagger do, they are far better than hard coded algorithms. SOTA has 97.96% of accuracy. In my opinion, having a world model (for common sense) and s…

they already use internally a POS tagger and a dependency parser. Interesting. Where did you see that?

Well it was just a belief. I may be wrong. I asked them by curiosity https://github.com/openai/gpt-2/issues/168 So we will know.

Re: GPT-2: 6-Month Follow-Up

#56

Post by AdamDKing »

minimaxir wrote: post #2

For finetuning GPT-2 on custom text, my gpt-2-simple package ( https://github.com/minimaxir/gpt-2-simple ) gets close to going OOM when finetuning the 345M model, even on a 16GB VRAM server GPU. Doubling the size of the model with the 774M model might cause it to not work at all, so I’ll need to test. Of course, the default output from the model might be sufficient, although it’ll take twice as long to generate text…

>How exactly the large GPT-2 models are deployed is a mystery I really wish was open-sourced more.

TalkToTransformer.com uses preemptible P4 GPUs on Google Kubernetes Engine. Changing the number of workers and automatically restarting them when they're preempted is easy with Kubernetes.

To provide outputs incrementally rather than waiting for the entire sequence to be generated, I open a websocket to a a worker and have it do a few tokens at a time, sending the output back as it goes. GPT-2 tokens can end partway through a multi-byte character, so to make this work you need to send the raw UTF-8 bytes to the browser and then have it concatenate them _before_ decoding the string.

While my workers can batch requests from multiple users, the modest increase in performance is probably not worth the complexity in most cases.

Re: GPT-2: 6-Month Follow-Up

#57

Post by gwern »

newhaus1994 wrote: post #52
gwern wrote: post #4

Earlier quoted context omitted.

I've already tried training with nshepperd's codebase. Sampling works, but even with the memory checkpointing and freezing the embedding and using SGD rather than Adam, it OOMs on a 1080ti's 11GB. Either additional tricks or CPU training are going to be required.

I'm the lead researcher on the Middlebury Institute project looking at fine-tuning the bigger models, and I got OOM on 745M and 1.5B originally. I had to get an Azure instance with 24GB VRAM to handle it (using nshepperd's codebase). It works, but takes a while (~500 epochs takes 12 hours on a 100k word training dataset).

Ouch! So 11GB is nowhere close to being enough, then. I wonder if even switching to FP16 will be adequate?

Re: GPT-2: 6-Month Follow-Up

#59

Post by p1esk »

the8472 wrote: post #31
minimaxir wrote: post #2

For finetuning GPT-2 on custom text, my gpt-2-simple package ( https://github.com/minimaxir/gpt-2-simple ) gets close to going OOM when finetuning the 345M model, even on a 16GB VRAM server GPU. Doubling the size of the model with the 774M model might cause it to not work at all, so I’ll need to test. Of course, the default output from the model might be sufficient, although it’ll take twice as long to generate text…

Possibly a stupid question, but does AMD lift such restrictions on models with its unified memory, by allowing the GPU to "page out" chunks of vram to system ram?

My guess is it would be much slower, because GPU processor would wait for data. Compare bandwidth - system RAM to GPU memory (PCIe): 16GBps vs GPU memory to GPU processor: 900GBps.

Re: GPT-2: 6-Month Follow-Up

#60

Post by ivalm »

The_rationalist wrote: post #36
make3 wrote: post #8

Earlier quoted context omitted.

BERT and its descendants do better at all of this, and are the industry standard now https://arxiv.org/abs/1810.04805

Except that BERT is now obscoleted by https://github.com/zihangdai/xlnet (but xlnet would never have existed without BERT)

Kind of, there are a bunch of transformers that might perform better than BERT (Ernie 2.0 being stronger than xlnet, for example), but often this is a function of training size (xlnet trained on 10x more data than original BERT). Realistically there are now BERTs released finetuned for special corpa (biobert, clinical bert, etc) so if you want to work on those kind of texts you are better off starting with a BERT that was previously fine tuned to something close to your task (and then fine tune it more yourself).
Post reply on HN