Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-2: 6-Month Follow-Up

openai.com

Post reply on HN
Print view View original

41–50 of 98 posts

Re: GPT-2: 6-Month Follow-Up

#41

Post by JonathanFly »

gwern wrote: post #27
JonathanFly wrote: post #24

Earlier quoted context omitted.

On the NVIDIA GPT- 2 implementation: >What would be the largest model one could train across 2x 2080Ti? >~800M gpt2. this is largely due to the memory required to House parameters + optimizer states. If one uses a smaller optimizer than Adam training something larger should be possible. Make sure to turn on activation checkpointing with —checkpoint-activations https://twitter.com/TheRealRPuri/status/11613225801264701…

They haven't released such models, though, and I don't know if it would be drop-in compatible with the OA GPT-2-774M checkpoint (they're training their own GPT-2s using their own webtext corpus).

I haven't look into at all myself, but he also said:

>We do provide training code that should work out of the box for gpt2 117M/345M

https://twitter.com/TheRealRPuri/status/1161319745259393024

Re: GPT-2: 6-Month Follow-Up

#42

Post by minimaxir »

gambler wrote: post #22

Hopefully someone will make a working demo of it, like Adam King did for 345M. People should be able to experiment with this stuff without relying on the hype of press releases: https://medium.com/@VictorBanev/interrogating-gpt-2-345m-aaf... Not sure why open AI doesn't do this themselves. That fully aligns with their stated mission.

It appears TalkToTransformer has been updated for 774M: https://twitter.com/AdamDanielKing/status/116387950071694131...

Re: GPT-2: 6-Month Follow-Up

#43

Post by p1esk »

The_rationalist wrote: post #34
p1esk wrote: post #21

Earlier quoted context omitted.

It's not to me to fixe that There's nothing to fix. People work on what they want to work on. Things that seem important to you are not important to me, and the opposite. I'm OK with that.

"People work on what they want to work" ideally yes, but ultimately they work on something that please them AND that give them a decent salary. Funding should not go to fun (but useless in the real world) Nlp tasks. "Things that seem important to you are not important to me, and the opposite." and here's go relativism or the abandon of thought... It's indeed difficult to quantify cardinally the utility of an NLP task…

do you understand that POS tagging or dependency/constictuency parsing are angular tasks needed by much of the others.

I'm not sure. I rarely have to do that explicitly in my head. Perhaps a model should learn to infer/guess them implicitly, from context, just like I do.

what exactly is more important?

In my opinion, having a world model (for common sense) and situational awareness (e.g. through sensor fusion, or from prior conversational history, or using some externally supplied conditioning) would be far more important.

GPT-2 does not generate meaningful sentences or only by accident

You think adding POS tags would help it generate meaningful sentences?

Re: GPT-2: 6-Month Follow-Up

#44

Post by p1esk »

JonathanFly wrote: post #41
gwern wrote: post #27

Earlier quoted context omitted.

They haven't released such models, though, and I don't know if it would be drop-in compatible with the OA GPT-2-774M checkpoint (they're training their own GPT-2s using their own webtext corpus).

I haven't look into at all myself, but he also said: >We do provide training code that should work out of the box for gpt2 117M/345M https://twitter.com/TheRealRPuri/status/1161319745259393024

It would take forever (or $$$) to train even 117M model from scratch.

Re: GPT-2: 6-Month Follow-Up

#45

Post by The_rationalist »

p1esk wrote: post #43
The_rationalist wrote: post #34

Earlier quoted context omitted.

"People work on what they want to work" ideally yes, but ultimately they work on something that please them AND that give them a decent salary. Funding should not go to fun (but useless in the real world) Nlp tasks. "Things that seem important to you are not important to me, and the opposite." and here's go relativism or the abandon of thought... It's indeed difficult to quantify cardinally the utility of an NLP task…

do you understand that POS tagging or dependency/constictuency parsing are angular tasks needed by much of the others. I'm not sure. I rarely have to do that explicitly in my head. Perhaps a model should learn to infer/guess them implicitly, from context, just like I do. what exactly is more important? In my opinion, having a world model (for common sense) and situational awareness (e.g. through sensor fusion, or fro…

I'm not sure. I rarely have to do that explicitly in my head. Well I can't prove it but I strongly believe that our brains use part of speech too, unconsciously. Perhaps a model should learn to infer/guess them implicitly, from data. That's exactly what deep learning POS tagger do, they are far better than hard coded algorithms. SOTA has 97.96% of accuracy.

In my opinion, having a world model (for common sense) and situational awareness (e.g. through sensor fusion, or from prior conversational history, or using some externally supplied conditioning) would be far more important. Haha you basically want a general intelligence (AGI), I want it too! And not enough persons works on "architecting" such a thing. Opencog may interest you a lot then. But the reality is many other "simpler" tasks are needed to make this happen.

having a world model (for common sense) is an NLP task There are some interesting results https://github.com/sebastianruder/NLP-progress/blob/master/e... OpenAI does not work on this task sadly, at least for now.

You think adding POS tags would help it generate meaningful sentences? I would be clearly insufficient yet necessary. I believe they already use internally a POS tagger and a dependency parser.

Re: GPT-2: 6-Month Follow-Up

#46

Post by JonathanFly »

p1esk wrote: post #44
JonathanFly wrote: post #41

Earlier quoted context omitted.

I haven't look into at all myself, but he also said: >We do provide training code that should work out of the box for gpt2 117M/345M https://twitter.com/TheRealRPuri/status/1161319745259393024

It would take forever (or $$$) to train even 117M model from scratch.

I read that meaning you can start with the actual pre-trained GPT-2 models but I never got an answer when I specifically asked if that was the case.

Re: GPT-2: 6-Month Follow-Up

#47

Post by comradesmith »

gambler wrote: post #28

"Cornell University is studying human susceptibility to digital disinformation generated by language models." "The Middlebury Institute of International Studies Center on Terrorism, Extremism, and Counterterrorism (CTEC) is exploring how GPT-2 could be misused by terrorists and extremists online." "The University of Oregon is developing a series of “bias probes” to analyze bias within GPT-2." But apparently no univer…

http://talktotransformer.com

Seems accessible to me

Re: GPT-2: 6-Month Follow-Up

#48

Post by dgellow »

the8472 wrote: post #32
The_rationalist wrote: post #7

Are there any real use case for GPT-2? Does it solve any problem? I've read almost all state of the art leaderboards of all Nlp tasks of paperswithcode.com and truth is except text generation, openAI has not one state of the art, they are not even visible in leaderboards. OpenAI is maybe the AI research center with the biggest funding and comparatively to other well known (Microsoft, Facebook, Google or even zalando.…

The article mentions TabNine, which has made it to the HN frontpage before.

For reference, the blog article for the release of "Deep TabNine", the auto-completion engine based on GPT-2: https://www.tabnine.com/blog/deep/.

And earlier this month they released a local version: https://www.tabnine.com/blog/local/.

Re: GPT-2: 6-Month Follow-Up

#50

Post by [deleted] »

lxe wrote: post #49

Anyone wired a "talktotransformer"-style system to this one yet? Would like to see how it works without going through the steps of setting it up. EDIT: Looks like https://talktotransformer.com/ already uses the 774M one!

[deleted]
Post reply on HN