Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-Neo – Building a GPT-3-sized model, open source and free

eleuther.ai

Post reply on HN
Print view View original

221–230 of 260 posts

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#221

Post by mitjam »

sailingparrot wrote: post #215
mirekrusin wrote: post #108

I'd gladly contribute (power and) few of idle GTX cards I have to public peer/volunteer/seti@home-like project if result snapshot(s) are available publicly/to registered, active contributors.

SETI@home style distributed computation is not suitable for training something like GPT-3, unlike for SETI, the unit of work a node can do before needing to share it's output with the next node is really small, so very fast interconnect between the nodes is needed (Infiniband and NVLink is used in clusters to train it). It would probably take a decade to train such a model over regular internet.

Maybe a case for a community colocation cloud where I a consumer can buy a system and colocate it in a large data center with great internal networking. Edit: typo

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#222

Post by leogao »

mitjam wrote: post #221
sailingparrot wrote: post #215

Earlier quoted context omitted.

SETI@home style distributed computation is not suitable for training something like GPT-3, unlike for SETI, the unit of work a node can do before needing to share it's output with the next node is really small, so very fast interconnect between the nodes is needed (Infiniband and NVLink is used in clusters to train it). It would probably take a decade to train such a model over regular internet.

Maybe a case for a community colocation cloud where I a consumer can buy a system and colocate it in a large data center with great internal networking. Edit: typo

Handling heterogenous (and potentially untrustworthy) systems also adds overhead, not to mention that buying hardware in bulk is cheaper, so it makes the most sense just to raise the money and buy the hardware.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#223

Post by colobas »

danwills wrote: post #26
newswasboring wrote: post #2

The intention behind it is pretty good. Best of luck to them. I wonder if I can donate computing power to this remotely. Like the old SETI or protein folding things. Use idle CPU to calculate for the network. Otherwise the estimates I have seen on how much it would take to train these models are enormous.

Yeah! sounds great, if I could easily run a SETI-at-home style thing to contribute to training a (future) model similar to GPT-x, but with the result freely available to play with, I reckon i'd do it. Could even be made into a game I'd say! I am totally aware that gpt3 itself can't be run on a regular workstation, but maybe hosting for instances of models from the best/most-interesting training runs could be worked o…

I was just gonna propose something like this. Democratizing large ML models.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#225

Post by kortex »

hntrader wrote: post #198
Filligree wrote: post #140

Earlier quoted context omitted.

It turns out that brute-force works, and the scaling curve is still not bending. I doubt we'll ever see a GPT-4, because there are known improvements they could make besides just upsizing it further, but that's besides the point. If that curve doesn't bend soon then a 10x larger network would be human-level in many ways. (Well, that is to say. It's actually bending. Upwards.)

What % of all digitized and reasonably easy-to-access text data did they use to train GPT-3? I'm wondering whether the current limits on GPT-n are computation or data.

> As per the creators, the OpenAI GPT-3 model has been trained about 45 TB text data from multiple sources which include Wikipedia and books.

It's about 400 B tokens. Library if Congress is about 40M books, let's say 50K tokens per book, or about 2T tokens. Not necessarily unique.

I would say it's plausible that it was a decent percent of the indexed text available, and even more of the unique content. GPT2 was 10B tokens. Do we have 20T tokens available for GPT4? Maybe. But the low hanging fruit are definitely plucked.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#226

Post by jne2356 »

pmontra wrote: post #63
techlatest_net wrote: post #54

Is there any real justification behind this fear of close nature of OpenAI or this is just frustration coming out? We had this debate of closed Vs open source 20 years back and eventually opensource won it because of various reasons. Won't those same reasons apply to this situation of close nature of OpenAI? If so then why are people worried about this? What is differnt this time?

The cost. Closed source and open source developers use the same $300-3,000 laptops / desktops. Everybody can afford them. Training a large model in a reasonable time costs much more. According to https://lambdalabs.com/blog/demystifying-gpt-3/ the cost of training GPT-3 was $4.6 million. Multiply it by the number of trial and errors. Of course we can't expect that something that costs tens or hundreds of millions wil…

OpenAI only trained the full sized GPT-3 once. Hyperparameter sweep was conducted on significantly smaller models (see: https://arxiv.org/abs/2001.08361)

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#227

Post by jne2356 »

Havoc wrote: post #78

Would be good if this could decentralized bittorrent/BOINC style somehow. Wouldn't mind contributing some horsepower

They get this suggestion a lot. There's a section in their FAQ that explains why it's infeasible.

https://github.com/EleutherAI/info

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#228

Post by jne2356 »

fakedang wrote: post #185
rexreed wrote: post #150

Once Microsoft became the sole and exclusive licensee of GPT-3, a credible open source effort was bound to emerge, and I hope it really is an effective alternative.

Eleuther works with Google. I'd rather prefer the Microsoft demon rather than the Google Demon.

The resources for the GPT-3 replication will be provided by the cloud company CoreWeave.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#230

Post by kashyapc »

Incidentally, over the weekend I listened to a two-hour-long presentation[1] by the cognitive scientist, Mark Turner[2]. The talk's main goal was to explore the question "Does cognitive linguistics offer a way forward for second-language teaching?"

During the Q&A, Turner explicitly mentions GPT-3 (that's when I first heard of it) as a "futuristic (but possible) language-learning environment" that is likely to be a great boon for second-language learners. One of the appealing points seems to be "conversation [with GPT-3] is not scripted; it keeps going on any subject you like". Thus allowing you to simulate some bits of the gold standard (immersive language learning in the real world).

As an advanced Dutch learner (as my fourth language), I'm curios of these approaches. And glad to see this open source model. (It is beyond ironic that the so-called "Open AI" turned out to have dubious ethics.)

[1] https://www.youtube.com/watch?v=A4Q977p8PfQ

[2] Check out the excellent book he co-authored, "Clear and simple as the truth"—it has valuable insights on improving writing based on some robust research.

Post reply on HN