Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-Neo – Building a GPT-3-sized model, open source and free

eleuther.ai

Post reply on HN
Print view View original

251–260 of 260 posts

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#251

Post by rijoja »

newswasboring wrote: post #2

The intention behind it is pretty good. Best of luck to them. I wonder if I can donate computing power to this remotely. Like the old SETI or protein folding things. Use idle CPU to calculate for the network. Otherwise the estimates I have seen on how much it would take to train these models are enormous.

You're a genius! How do we get this started. Wouldn't there be so many bitcoin rigs already that could be used for this already?

I'd be very willing to spend time and money on this!

Could someone set up a chat room or something?

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#252

Post by stellaathena »

aabhay wrote: post #161
pjfin123 wrote: post #159

What does the future of open-source large neural nets look like? My understanding is GPT-3 takes ~600GB of GPU memory to run inference. Does an open source model just allow you a choice of a handful of cloud providers instead of one?

Open source doesn’t mean that everyone will be rolling their own. It means that lots of players will start to offer endpoints with GPT-X, perhaps bundled with other services. It is good for the market.

Right, this is why a cloud company is sponsoring the replication. They can’t sell GPT-3 as a service until they get a copy of it.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#253

Post by stellaathena »

qayxc wrote: post #100
pmontra wrote: post #63

Earlier quoted context omitted.

The cost. Closed source and open source developers use the same $300-3,000 laptops / desktops. Everybody can afford them. Training a large model in a reasonable time costs much more. According to https://lambdalabs.com/blog/demystifying-gpt-3/ the cost of training GPT-3 was $4.6 million. Multiply it by the number of trial and errors. Of course we can't expect that something that costs tens or hundreds of millions wil…

This. Plus the increasing amount of intransparent results. Training data is private, so it's impossible to even try to recreate results, validate methods, or find biases/failure cases.

This is dead right and very important. The data you train on is much more important than model architecture in terms of validation and compliance, and yet it’s a closely held secret. Producing or obtaining good data is a pain in the ass.

For this reason, EAI has made the data we are training on public. I can’t link to it because of anon policies at conferences, but if you look at our website I’m sure you can find a paper detailing it and a link to download it.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#254

Post by stellaathena »

fakedang wrote: post #185
rexreed wrote: post #150

Once Microsoft became the sole and exclusive licensee of GPT-3, a credible open source effort was bound to emerge, and I hope it really is an effective alternative.

Eleuther works with Google. I'd rather prefer the Microsoft demon rather than the Google Demon.

“Works with Google” and “gets free compute from TFRC” are not really the same thing.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#255

Post by stellaathena »

anticristi wrote: post #3

I love the initiative, but I'm starting to get scared of what a post-GPT-3 world will look like. We are already struggling to distinguish fake news from real ones, automated customer request replies from genuine replies, etc. How will I know that I have a conversation with a real human in the future? On the other side, the prospect of having an oracle that answers all trivia, fixes spelling and grammar, and allows hu…

It’s a lot less expensive to hire a dozen teenagers to write fake news than use GPT-3. The cost to train GPT-3 is about the same as the cost to hire 1,000 people and pay them $10/hour to write responses for a year. Not to mention the fact that inference isn’t free.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#256

Post by stellaathena »

gianlucahmd wrote: post #43
anticristi wrote: post #3

I love the initiative, but I'm starting to get scared of what a post-GPT-3 world will look like. We are already struggling to distinguish fake news from real ones, automated customer request replies from genuine replies, etc. How will I know that I have a conversation with a real human in the future? On the other side, the prospect of having an oracle that answers all trivia, fixes spelling and grammar, and allows hu…

We already live in a post-GPT3 world, but one where all its power is the hands of a private company. The conversation needs to move on whether making it open and democratic is a good idea, but the tech itself is here to stay.

EleutherAI’s unofficial motto is “anything safe enough for Microsoft to sell for profit is safe enough to release.” You’re kidding yourself if you think Microsoft cares who they sell it to.

There are worthwhile conversations to have about democratizing technology, but this stuff is already out there regardless of what you or I do.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#257

Post by woko »

kordlessagain wrote: post #232
typon wrote: post #216

Earlier quoted context omitted.

For training yes, but not for inference.

From 2019: https://heartbeat.fritz.ai/deep-learning-has-a-size-problem-... > Earlier this year, researchers at NVIDIA announced MegatronLM, a massive transformer model with 8.3 billion parameters (24 times larger than BERT) > The parameters alone weigh in at just over 33 GB on disk. Training the final model took 512 V100 GPUs running continuously for 9.2 days. Running this model on a "regular" machine at some useful…

Sorry, but I don't see the link between the quote and your sentence.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#259

Post by Havoc »

jne2356 wrote: post #227
Havoc wrote: post #78

Would be good if this could decentralized bittorrent/BOINC style somehow. Wouldn't mind contributing some horsepower

They get this suggestion a lot. There's a section in their FAQ that explains why it's infeasible. https://github.com/EleutherAI/info

I see. THanks for highlighting that. Makes sense unfortunately
Post reply on HN