GPT-Neo – Building a GPT-3-sized model, open source and free • Hacker News Board

GPT-Neo – Building a GPT-3-sized model, open source and free

#1

Post by sieste » Mon, Jan 18, 2021, 9:13 AM UTC

GPT-Neo – Building a GPT-3-sized model, open source and free
eleuther.ai

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#2

Post by newswasboring » Mon, Jan 18, 2021, 9:36 AM UTC

The intention behind it is pretty good. Best of luck to them.

I wonder if I can donate computing power to this remotely. Like the old SETI or protein folding things. Use idle CPU to calculate for the network. Otherwise the estimates I have seen on how much it would take to train these models are enormous.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#3

Post by anticristi » Mon, Jan 18, 2021, 9:39 AM UTC

I love the initiative, but I'm starting to get scared of what a post-GPT-3 world will look like. We are already struggling to distinguish fake news from real ones, automated customer request replies from genuine replies, etc. How will I know that I have a conversation with a real human in the future?

On the other side, the prospect of having an oracle that answers all trivia, fixes spelling and grammar, and allows humans to focus on higher level information processing is interesting.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#4

Post by Erlich_Bachman » Mon, Jan 18, 2021, 9:40 AM UTC

It's a shame that it has turned out to be necessary to externally re-make and re-train a model that has come out of company called `OPEN`AI. Wasn't one of the founding principles of it that all of the research would be available to the public? Isn't that the premise on which the initial funding was secured? Best of luck to Eleuther.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#5

Post by teruakohatu » Mon, Jan 18, 2021, 9:41 AM UTC

newswasboring wrote: post #2

The intention behind it is pretty good. Best of luck to them. I wonder if I can donate computing power to this remotely. Like the old SETI or protein folding things. Use idle CPU to calculate for the network. Otherwise the estimates I have seen on how much it would take to train these models are enormous.

TensorFlow supports distributed training with a client server model.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#6

Post by Erlich_Bachman » Mon, Jan 18, 2021, 9:42 AM UTC

anticristi wrote: post #3

I love the initiative, but I'm starting to get scared of what a post-GPT-3 world will look like. We are already struggling to distinguish fake news from real ones, automated customer request replies from genuine replies, etc. How will I know that I have a conversation with a real human in the future? On the other side, the prospect of having an oracle that answers all trivia, fixes spelling and grammar, and allows hu…

Photoshop has existed for decades. Is it really that big of a problem for photo news?

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#7

Post by nemoniac » Mon, Jan 18, 2021, 9:43 AM UTC

Good initiative but tell us more about the governance. After all OpenAI was "open" until it was bought by Microsoft.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#8

Post by spiderfarmer » Mon, Jan 18, 2021, 9:44 AM UTC

Erlich_Bachman wrote: post #4

It's a shame that it has turned out to be necessary to externally re-make and re-train a model that has come out of company called `OPEN`AI. Wasn't one of the founding principles of it that all of the research would be available to the public? Isn't that the premise on which the initial funding was secured? Best of luck to Eleuther.

I don't really care if OpenAI offers commercial licenses as long as the underlying research is truly open. This way alternative options will become available eventually.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#9

Post by turing_complete » Mon, Jan 18, 2021, 9:44 AM UTC

anticristi wrote: post #3

I love the initiative, but I'm starting to get scared of what a post-GPT-3 world will look like. We are already struggling to distinguish fake news from real ones, automated customer request replies from genuine replies, etc. How will I know that I have a conversation with a real human in the future? On the other side, the prospect of having an oracle that answers all trivia, fixes spelling and grammar, and allows hu…

Don't read news. Go to original sources and scientific papers. If you really want to understand something, a news website should only be your starting point to look for keywords. That is true today as it will be "post-GPT-3".

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#10

Post by 2Gkashmiri » Mon, Jan 18, 2021, 9:46 AM UTC

So how much money would it take to rebuild this foss alternative ? And distributive power like seti@home? If it can be done and I hope it does, what benefit would the original proprietary one have over this? Licensing?