Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-Neo – Building a GPT-3-sized model, open source and free

eleuther.ai

Post reply on HN
Print view View original

171–180 of 260 posts

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#171

Post by thelastwave »

dave_sullivan wrote: post #48
Erlich_Bachman wrote: post #4

It's a shame that it has turned out to be necessary to externally re-make and re-train a model that has come out of company called `OPEN`AI. Wasn't one of the founding principles of it that all of the research would be available to the public? Isn't that the premise on which the initial funding was secured? Best of luck to Eleuther.

Really. OpenAI assembled some of the best minds from the deep learning community. The problem isn't that they are a for-profit SaaS, the problem is they lied.

And ended up making an AI service that's really good at... lying.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#172

Post by jokowueu »

Mizza wrote: post #123

Serious question: is there a warez scene for trained models yet? (I don't know how the model is accessed - are users of mainline GPT-3 given a .pb and a stack of NDAs, or do they have to access it through access-controlled API?) Wherever data is desired by many but held by a few, a pirate crew inevitably emerges.

I think this also might be an interest to you

https://the-eye.eu/public/AI/pile_preliminary_components/

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#173

Post by jariel »

Sambdala wrote: post #73
Erlich_Bachman wrote: post #4

It's a shame that it has turned out to be necessary to externally re-make and re-train a model that has come out of company called `OPEN`AI. Wasn't one of the founding principles of it that all of the research would be available to the public? Isn't that the premise on which the initial funding was secured? Best of luck to Eleuther.

Wild-Ass Guess (Ass-Guess) incoming: OpenAI was built to influence the eventual value chain of AI in directions that would give the funding parties more confidence that their AI bets would pay off. This value chain basically being one revolving around AI as substituting predictions and human judgement in a business process, much like cloud can be (oversimply) modeled as moving Capex to Opex in IT procurement. They sa…

This is neat, but almost no startups of any kind, even mid size corps, have such complicated and intricate plans.

More likely: OpenAI was a legit premise, they started to run out of money, MS wanted to license and it wasn't going to work otherwise, so they just took the temperature with their initial sponsors and staff and went commercial.

And that's it.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#174

Post by jne2356 »

newswasboring wrote: post #2

The intention behind it is pretty good. Best of luck to them. I wonder if I can donate computing power to this remotely. Like the old SETI or protein folding things. Use idle CPU to calculate for the network. Otherwise the estimates I have seen on how much it would take to train these models are enormous.

They get this suggestion a lot. There's a section in their FAQ that explains why it's infeasible.

https://github.com/EleutherAI/info

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#175

Post by jne2356 »

Filligree wrote: post #138
emteycz wrote: post #85

Earlier quoted context omitted.

What about some kind of sharding, parts of the computation that could be executed in isolation for a longer period of time?

An ongoing research problem. OpenAI would certainly like being able to use smaller GPUs, instead of having to fit the entire model into one.

GPT-3 does not fit in any one GPU that exists at present. It's already spread out across multiple GPUs.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#177

Post by thelastwave »

tmalsburg2 wrote: post #81
joshlk wrote: post #69

How does the outfit intend to fund the project? OpenAI spends millions on computing resources to train the models.

I imagine recreating the model will be computationally cheaper because they will not have to sift through the same huge hyperparameter space as the initial GPT-3 team had to.

Why is that?

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#179

Post by jne2356 »

2Gkashmiri wrote: post #10

So how much money would it take to rebuild this foss alternative ? And distributive power like seti@home? If it can be done and I hope it does, what benefit would the original proprietary one have over this? Licensing?

EleutherAI has already secured the resources necessary.

They get the seti@home suggestion a lot. There's a section in their FAQ that explains why it's infeasible.

https://github.com/EleutherAI/info

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#180

Post by jne2356 »

tmalsburg2 wrote: post #81
joshlk wrote: post #69

How does the outfit intend to fund the project? OpenAI spends millions on computing resources to train the models.

I imagine recreating the model will be computationally cheaper because they will not have to sift through the same huge hyperparameter space as the initial GPT-3 team had to.

This is not true. The OpenAI team only trained one full-sized GPT-3, and conducted their hyperparameter sweep on significantly smaller models (see: https://arxiv.org/abs/2001.08361). The compute savings from not having to do the hyperparameter sweep are negligible and do not significantly change the feasibility of the project.
Post reply on HN