Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-Neo – Building a GPT-3-sized model, open source and free

eleuther.ai

Post reply on HN
Print view View original

41–50 of 260 posts

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#41

Post by person_of_color »

wraptile wrote: post #19
Erlich_Bachman wrote: post #4

It's a shame that it has turned out to be necessary to externally re-make and re-train a model that has come out of company called `OPEN`AI. Wasn't one of the founding principles of it that all of the research would be available to the public? Isn't that the premise on which the initial funding was secured? Best of luck to Eleuther.

OpenAI turning out to be a total bait and switch. Especially true when your co-founder is actively calling you out on it[1] Remember kids: if it's not a non-profit organization it is a _for_ profit one! It was silly to expect anything else: > In 2019, OpenAI transitioned from non-profit to for-profit. The company distributed equity to its employees and partnered with Microsoft Corporation, who announced an investment…

So OpenAI employees get Microsoft RSUs?

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#42

Post by newswasboring »

teruakohatu wrote: post #5
newswasboring wrote: post #2

The intention behind it is pretty good. Best of luck to them. I wonder if I can donate computing power to this remotely. Like the old SETI or protein folding things. Use idle CPU to calculate for the network. Otherwise the estimates I have seen on how much it would take to train these models are enormous.

TensorFlow supports distributed training with a client server model.

Does it also solve the problem of everyone having different hardware?

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#43

Post by gianlucahmd »

anticristi wrote: post #3

I love the initiative, but I'm starting to get scared of what a post-GPT-3 world will look like. We are already struggling to distinguish fake news from real ones, automated customer request replies from genuine replies, etc. How will I know that I have a conversation with a real human in the future? On the other side, the prospect of having an oracle that answers all trivia, fixes spelling and grammar, and allows hu…

We already live in a post-GPT3 world, but one where all its power is the hands of a private company.

The conversation needs to move on whether making it open and democratic is a good idea, but the tech itself is here to stay.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#44

Post by dkjaudyeqooe »

anticristi wrote: post #3

I love the initiative, but I'm starting to get scared of what a post-GPT-3 world will look like. We are already struggling to distinguish fake news from real ones, automated customer request replies from genuine replies, etc. How will I know that I have a conversation with a real human in the future? On the other side, the prospect of having an oracle that answers all trivia, fixes spelling and grammar, and allows hu…

I know! One day it's going to get so bad people are going to have to deploy critical thinking instead of accepting what they read at face value and suffer the indignity of having to think for themselves.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#45

Post by londons_explore »

newswasboring wrote: post #42
teruakohatu wrote: post #5

Earlier quoted context omitted.

TensorFlow supports distributed training with a client server model.

Does it also solve the problem of everyone having different hardware?

It does.

For most models, your home broadband would be far too slow though.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#46

Post by xerxespoy »

phreeza wrote: post #31
taneq wrote: post #14

Earlier quoted context omitted.

The fake news thing is a real problem (and may become worse under GPT3 but certainly exists already). As for the others - to quote Westworld, "if you can't tell the difference, does it really matter?"

Genuine question, why is this a problem? Sure, someone may be able to generate thousands of real-sounding fake news articles, but it's not like they will also be able to flood the New York Times with these articles. How do you worry you will be exposed to these articles?

Journalists are paid by the word.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#47

Post by unixhero »

person_of_color wrote: post #41
wraptile wrote: post #19

Earlier quoted context omitted.

OpenAI turning out to be a total bait and switch. Especially true when your co-founder is actively calling you out on it[1] Remember kids: if it's not a non-profit organization it is a _for_ profit one! It was silly to expect anything else: > In 2019, OpenAI transitioned from non-profit to for-profit. The company distributed equity to its employees and partnered with Microsoft Corporation, who announced an investment…

So OpenAI employees get Microsoft RSUs?

What is an RSU?

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#48

Post by dave_sullivan »

Erlich_Bachman wrote: post #4

It's a shame that it has turned out to be necessary to externally re-make and re-train a model that has come out of company called `OPEN`AI. Wasn't one of the founding principles of it that all of the research would be available to the public? Isn't that the premise on which the initial funding was secured? Best of luck to Eleuther.

Really. OpenAI assembled some of the best minds from the deep learning community. The problem isn't that they are a for-profit SaaS, the problem is they lied.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#49

Post by chillee »

newswasboring wrote: post #2

The intention behind it is pretty good. Best of luck to them. I wonder if I can donate computing power to this remotely. Like the old SETI or protein folding things. Use idle CPU to calculate for the network. Otherwise the estimates I have seen on how much it would take to train these models are enormous.

The primary issue is that large scale GPU training is primarily dominated by communication costs. Since to some approximation things need to be synchronized after every gradient update, it becomes very quickly quite infeasible to increase the communication cost.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#50

Post by gmueckl »

dkjaudyeqooe wrote: post #44
anticristi wrote: post #3

I love the initiative, but I'm starting to get scared of what a post-GPT-3 world will look like. We are already struggling to distinguish fake news from real ones, automated customer request replies from genuine replies, etc. How will I know that I have a conversation with a real human in the future? On the other side, the prospect of having an oracle that answers all trivia, fixes spelling and grammar, and allows hu…

I know! One day it's going to get so bad people are going to have to deploy critical thinking instead of accepting what they read at face value and suffer the indignity of having to think for themselves.

Critical thinking won't help you when the majority (or all) of your sources are tainted and contradictory. At some point, the actual truth just gets swamped.
Post reply on HN