GPT-J: GPT-3 Democratized • Hacker News Board

Re: GPT-J: GPT-3 Democratized

#21

Post by mkl95 » Sun, Jul 04, 2021, 7:45 PM UTC

loosetypes wrote: post #20

Does anyone else take fault with the increasingly commonplace usage of the word democratize as applied to technology? For every valid case, I see others that make my head hurt. Is it just a buzzword for telling a story with emotional appeal to users and investors? Making something available to someone who didn’t have it earlier isn’t democratizing. And ignoring future considerations is just lazy. If guns were invente…

I had a History teacher who told me democracy often doesn't really mean anything. I've been thinking about it ever since. "Free GPT-3" would probably be better.

Re: GPT-J: GPT-3 Democratized

#22

Post by retrac » Sun, Jul 04, 2021, 7:45 PM UTC

c7DJTLrn wrote: post #16

zitterbewegung wrote: post #4

I run GPT-J on a Titan RTX where I am writing a novel with it. To make it generate about 20k tokens or two pages of content takes a few minutes . I would say the output is comparable to other language models quality and so forth. Note that refinement or transfer learning doesn’t apply anymore it’s more like using a zero shot classifier or in other words you have to craft the input like Siri or wolfram alpha but expec…

That seems ridiculously slow. How is machine learning supposed to scale like this?

I mean there's two ways.

1. Just wait a few years for better hardware. Not even joking on that one. The computational requirements of most modern GAN and transformer-based approaches, like the StyleGAN image generation I was playing around with recently, would have been daunting even on the largest supercomputers just 10 or 15 years ago and before that it was science fiction.

2. Find more shortcuts. We all knew neural nets could learn any function all along, in principle. If you have infinite time and space, some of the simplest multilayer feedback networks can learn anything learnable. I don't think anyone really expected to find such relatively efficient algorithms for that, like we have in the last few years, though.

Re: GPT-J: GPT-3 Democratized

#23

Post by robbedpeter » Sun, Jul 04, 2021, 7:47 PM UTC

mhuffman wrote: post #17

I was told that GPT-3 was too much power for mere mortals (without a paid subscription!) so what terror is this going to bring upon us?

Transparent and competent automated content moderation, maybe, easily available for anyone to run their own communities by their own standards. Once matured, you can easily envision people sharing policies and templates, or providing moderation as a service, for any sort of social text interaction.

Re: GPT-J: GPT-3 Democratized

#24

Post by hekec » Sun, Jul 04, 2021, 7:47 PM UTC

Love this!

Could anyone recommend a tutorial how to run this? I need to generate answers to 1000 questions for my app. Still waiting for GPT-3 invite for months

Re: GPT-J: GPT-3 Democratized

#25

Post by coolspot » Sun, Jul 04, 2021, 7:50 PM UTC

gurchik wrote: post #12

zitterbewegung wrote: post #4

I run GPT-J on a Titan RTX where I am writing a novel with it. To make it generate about 20k tokens or two pages of content takes a few minutes . I would say the output is comparable to other language models quality and so forth. Note that refinement or transfer learning doesn’t apply anymore it’s more like using a zero shot classifier or in other words you have to craft the input like Siri or wolfram alpha but expec…

> I am writing a novel with it. Could you say more?

Spam Amazon Kindle store with GPT-generated $.99 novel books.

Re: GPT-J: GPT-3 Democratized

#26

Post by captainmuon » Sun, Jul 04, 2021, 7:55 PM UTC

How much resources would it take to train something like this? Let's say in RTX-3090 months, or in $ running on AWS (or whoever offers TPUs)?

In the link it says training FLOPs: 1.5e22, a RTX 3090 has 285 TFLOPS/sec for tensor operations. If you can actually calculate it like that, 1000 GPUs would take 20 months.

https://www.google.com/search?q=1.5e22+%2F+%281000+*+285e9+%...

Working with grid computing at CERN, I've had access to some pretty big computing resources myself, but the scale of this is mind-boggling...

Re: GPT-J: GPT-3 Democratized

#27

Post by la6471 » Sun, Jul 04, 2021, 7:56 PM UTC

zitterbewegung wrote: post #4

I run GPT-J on a Titan RTX where I am writing a novel with it. To make it generate about 20k tokens or two pages of content takes a few minutes . I would say the output is comparable to other language models quality and so forth. Note that refinement or transfer learning doesn’t apply anymore it’s more like using a zero shot classifier or in other words you have to craft the input like Siri or wolfram alpha but expec…

However the fact that it is trained with 6B parameters compared to GPT-3’s 175B parameters indicate open source GPT still has a lot to catch on. GodSpeed and full speed !!!

Re: GPT-J: GPT-3 Democratized

#28

Post by EamonnMR » Sun, Jul 04, 2021, 8:01 PM UTC

robbedpeter wrote: post #23

mhuffman wrote: post #17

I was told that GPT-3 was too much power for mere mortals (without a paid subscription!) so what terror is this going to bring upon us?

Transparent and competent automated content moderation, maybe, easily available for anyone to run their own communities by their own standards. Once matured, you can easily envision people sharing policies and templates, or providing moderation as a service, for any sort of social text interaction.

I was just thinking about this today. How many (and which?) ML books would one need to read to be able to throw comments into a model like this and have it decide if they're acceptable vs unacceptable? How practical would it be to build a frequently re-training model that could serve as autopilot for a small forum or subreddit moderation? And how much would that serve to reinforce existing filter bubbles and further divide the internet?

Re: GPT-J: GPT-3 Democratized

#29

Post by loosetypes » Sun, Jul 04, 2021, 8:01 PM UTC

mkl95 wrote: post #21

loosetypes wrote: post #20

Does anyone else take fault with the increasingly commonplace usage of the word democratize as applied to technology? For every valid case, I see others that make my head hurt. Is it just a buzzword for telling a story with emotional appeal to users and investors? Making something available to someone who didn’t have it earlier isn’t democratizing. And ignoring future considerations is just lazy. If guns were invente…

I had a History teacher who told me democracy often doesn't really mean anything. I've been thinking about it ever since. "Free GPT-3" would probably be better.

Interesting.

I guess my off the cuff, unexamined criteria for democratizing is whether it’s a gating/enabling function for the occurrence of one party’s objectives as directly decided by the wider community.

So something like GoFundMe or crowdfunding, sure. Although doesn’t necessarily need to be monetary.

But I’d be interested in hearing counteropinions.

Re: GPT-J: GPT-3 Democratized

#30

Post by polynomial » Sun, Jul 04, 2021, 8:16 PM UTC

Tenoke wrote: post #6

rorykoehler wrote: post #3

Dupe https://news.ycombinator.com/item?id=27727009

Yeah, twice in one day (and 3+ times in a month) is a bit much and this post adds little new except some oddities like singling kubernetes in particular as 'the technology that helped train GPT-3'.

As the article notes, "There’s quite a similarity between cloud native and Artificial Intelligence."