Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-Neo – Building a GPT-3-sized model, open source and free

eleuther.ai

Post reply on HN
Print view View original

61–70 of 260 posts

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#61

Post by realusername »

dkjaudyeqooe wrote: post #44
anticristi wrote: post #3

I love the initiative, but I'm starting to get scared of what a post-GPT-3 world will look like. We are already struggling to distinguish fake news from real ones, automated customer request replies from genuine replies, etc. How will I know that I have a conversation with a real human in the future? On the other side, the prospect of having an oracle that answers all trivia, fixes spelling and grammar, and allows hu…

I know! One day it's going to get so bad people are going to have to deploy critical thinking instead of accepting what they read at face value and suffer the indignity of having to think for themselves.

I'm going to throw some wild guess here and say that this sudden increase in critical thinking won't happen.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#62

Post by ForHackernews »

mhuffman wrote: post #39
Erlich_Bachman wrote: post #4

It's a shame that it has turned out to be necessary to externally re-make and re-train a model that has come out of company called `OPEN`AI. Wasn't one of the founding principles of it that all of the research would be available to the public? Isn't that the premise on which the initial funding was secured? Best of luck to Eleuther.

But I was told GPT-3 was too powerful for mere mortal hands (unless you have an account!) and that it would be used for hate speech and to bring about skynet. How will this project avoid those terrible outcomes?

New research has revealed that intelligence is not a prerequisite for generating hate speech on social media platforms.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#63

Post by pmontra »

techlatest_net wrote: post #54

Is there any real justification behind this fear of close nature of OpenAI or this is just frustration coming out? We had this debate of closed Vs open source 20 years back and eventually opensource won it because of various reasons. Won't those same reasons apply to this situation of close nature of OpenAI? If so then why are people worried about this? What is differnt this time?

The cost.

Closed source and open source developers use the same $300-3,000 laptops / desktops. Everybody can afford them.

Training a large model in a reasonable time costs much more. According to https://lambdalabs.com/blog/demystifying-gpt-3/ the cost of training GPT-3 was $4.6 million. Multiply it by the number of trial and errors.

Of course we can't expect that something that costs tens or hundreds of millions will be given away for free or to be able to rebuild it without some collective training effort that distributes the cost on at least thousands of volunteers.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#64

Post by hnlmorg »

phreeza wrote: post #31
taneq wrote: post #14

Earlier quoted context omitted.

The fake news thing is a real problem (and may become worse under GPT3 but certainly exists already). As for the others - to quote Westworld, "if you can't tell the difference, does it really matter?"

Genuine question, why is this a problem? Sure, someone may be able to generate thousands of real-sounding fake news articles, but it's not like they will also be able to flood the New York Times with these articles. How do you worry you will be exposed to these articles?

If recent times have told us anything, it's that the biggest distributor of "news" is social media. And worse still, people generally have no interest in researching the items they read. If "fake news" confirms their pre-existing bias then they will automatically believe it. If real news disagrees with their biases then it is considered fake.

So in theory, the rise of deep fakes could lead to more people getting suckered into conspiracy theories and other such extreme opinions. We've already seen a small trend this way with low resolution images of different people with vaguely similar physical features because used as "evidence" of actors in hospitals / shootings / terrorist scenes / etc.

That all said, I don't see this as a reason not to pursue GPT-3. From that regard the proverbial genie is already out of the bottle. What we need to work on is a better framework for distributing knowledge.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#65

Post by newswasboring »

londons_explore wrote: post #45
newswasboring wrote: post #42

Earlier quoted context omitted.

Does it also solve the problem of everyone having different hardware?

It does. For most models, your home broadband would be far too slow though.

Is it because they will have to communicate back errors during training? I forgot that training these models is more of a global task than proteins folding. In that sense this is less parallelizable over the internet.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#66

Post by hooande »

In my experience, the output from GPT-3, DALL-E, et al is similar to what you get from googling the prompt and stitching together snippets from the top results. These transformers are trained on "what was visible to google", which provides the limitation on their utility.

I think of the value proposition of GPT-X as "what would you do with a team of hundreds of people who can solve arbitrary problems only by googling them?". And honestly, not a lot of productive applications come to mind.

This is basically the description of a modern content farm. You give N people a topic, ie "dating advice", and they'll use google to put together different ideas, sentences and paragraphs to produce dozens of articles per day. You could also write very basic code with this, similar to googling a code snippet and combining the results from the first several stackoverflow pages that come up (which, incidentally, is how I program now). After a few more versions, you could probably use GPT to produce fiction that matches the quality of the average self published ebook. And DALL-E can come up with novel images in the same way that a graphic designer can visually merge the google image results for a given query.

One limitation of this theoretical "team of automated googlers" is that what they search for content is cached on the date of the last GPT model update. Right now the big news story is the Jan 6th, 20201 insurrection at the US Capitol. GPT-3 can produce infinite bad articles about politics, but won't be able to say anything about current events in real time.

I generally think that GPT-3 is awesome, and it's a damn shame that "Open"AI couldn't find a way to actually be open. At this point, it seems like a very interesting technology that is still in desperate need of a Killer App

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#67

Post by visarga »

Erlich_Bachman wrote: post #4

It's a shame that it has turned out to be necessary to externally re-make and re-train a model that has come out of company called `OPEN`AI. Wasn't one of the founding principles of it that all of the research would be available to the public? Isn't that the premise on which the initial funding was secured? Best of luck to Eleuther.

It was probably bait and switch to hire top researchers and get initial funding. Now that OpenAI is a household name, they don't have to pretend anymore.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#68

Post by Pyramus »

darkwater wrote: post #60
IndySun wrote: post #53

Earlier quoted context omitted.

"one of the things that should absolutely be taught in school but isn't." Namely, critical thinking?

I don't think "critical thinking" is the point here. Because first you need to know that such modifications CAN be done. And not everybody knows what can be retouched with PS or programs. So yeah, if you see some super-model on a magazine cover, and you don't know PS can edit photos easily, it would be not that immediate to think "hey maybe that's not real!". As an extreme example: would you ever checked 20 years ago…

Exactly this.

There is a secondary aspect of becoming aware that society has agreed on beauty standards (different for different societies) and PS being used as a means to adhere to these standards.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#70

Post by inglor_cz »

gmueckl wrote: post #50
dkjaudyeqooe wrote: post #44

Earlier quoted context omitted.

I know! One day it's going to get so bad people are going to have to deploy critical thinking instead of accepting what they read at face value and suffer the indignity of having to think for themselves.

Critical thinking won't help you when the majority (or all) of your sources are tainted and contradictory. At some point, the actual truth just gets swamped.

This. Robots can spout 1000x more content than humans, if not more.
Post reply on HN