Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-Neo – Building a GPT-3-sized model, open source and free

eleuther.ai

Post reply on HN
Print view View original

91–100 of 260 posts

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#91

Post by littlestymaar »

b3kart wrote: post #71
visarga wrote: post #67

Earlier quoted context omitted.

It was probably bait and switch to hire top researchers and get initial funding. Now that OpenAI is a household name, they don't have to pretend anymore.

I buy the former, researchers might be happier knowing their work potentially benefits all of humanity, not just a bunch of investors. But wouldn’t it be more difficult to get funding as a non-profit?

It's just never going to be difficult to get funding when you have Elon Musk and Sam Altman as founders (and even more so when founders put one billion of their own money into it).

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#92

Post by qayxc »

taneq wrote: post #14
anticristi wrote: post #3

I love the initiative, but I'm starting to get scared of what a post-GPT-3 world will look like. We are already struggling to distinguish fake news from real ones, automated customer request replies from genuine replies, etc. How will I know that I have a conversation with a real human in the future? On the other side, the prospect of having an oracle that answers all trivia, fixes spelling and grammar, and allows hu…

The fake news thing is a real problem (and may become worse under GPT3 but certainly exists already). As for the others - to quote Westworld, "if you can't tell the difference, does it really matter?"

> "if you can't tell the difference, does it really matter?"

It indeed does. The problem is that societies and cultures are heavily influenced and changed by communication, media, and art.

By replacing big portions of these components with artificial content, generated from previously created content, you run the risk of creating feedback cycles (e.g. train future systems from output of their predecessors) and forming standards (beauty, aesthetics, morality, etc.) controlled by the entities that build, train, and filter the output of the AIs.

You'll basically run the risk of killing individuality and diversity in culture and expression; consequences on society as a whole and individual behaviour are difficult to predict, but seeing how much power social media (an unprecedented phenomenon in human culture) have, there's reason to at the very least be cautious about this.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#93

Post by chimprich »

phreeza wrote: post #31
taneq wrote: post #14

Earlier quoted context omitted.

The fake news thing is a real problem (and may become worse under GPT3 but certainly exists already). As for the others - to quote Westworld, "if you can't tell the difference, does it really matter?"

Genuine question, why is this a problem? Sure, someone may be able to generate thousands of real-sounding fake news articles, but it's not like they will also be able to flood the New York Times with these articles. How do you worry you will be exposed to these articles?

It's not me I'm worried about - it's the 50% [1] of people who get their news from social media and "entertainment" news platforms. These people vote, and can get manipulated into performing quite extreme acts.

At the moment a lot of people seem to have trouble engaging with reality, and that seems to be caused by relatively small disinformation campaigns and viral rumours. How much worse could it get when there's a vast number of realistic-sounding news articles appearing, accompanied by realistic AI-generated photos and videos?

And that might not even be the biggest problem. If these things can be generated automatically and easily, it's going to be very easy to dismiss real information as fake. The labelling of real news as "fake news" phenomenon is going to get bigger.

It's going to be more work to distinguish what is real from what is fake. If it's possible to find articles supporting any position and a suspicion that any contrary new is then a lot of people are going to find it easier to just believe what they prefer to believe... even more than they do now.

[1] made-up number, but doesn't feel far off.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#94

Post by JacobiX »

visarga wrote: post #74
JacobiX wrote: post #20

Earlier quoted context omitted.

It's a recurring theme in OpenAI research, they become more and more closed. For instance their latest model called DALL·E hit the headlines before the release of the paper. Needless to say, the model is not available and no online demo has been published so far.

But they still released the CLIP model which is the complement of DALL-E and used in the DALL-E pipeline as a final filter. There are collabs with CLIP floating around and even a web demo.

Thank you for this info, as you mentioned CLIP is used for re-ranking DALL-E outputs, by itself it is just an (image, text) pairs classification network.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#95

Post by visarga »

hooande wrote: post #66

In my experience, the output from GPT-3, DALL-E, et al is similar to what you get from googling the prompt and stitching together snippets from the top results. These transformers are trained on "what was visible to google", which provides the limitation on their utility. I think of the value proposition of GPT-X as "what would you do with a team of hundreds of people who can solve arbitrary problems only by googling…

> honestly, not a lot of productive applications come to mind

Not so convincing when you enumerate so many applications yourself.

> but won't be able to say anything about current events

There are variants that use transformer + retrieval, so they got unlimited memory that can be easily extended.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#96

Post by profunctor »

chimprich wrote: post #93
phreeza wrote: post #31

Earlier quoted context omitted.

Genuine question, why is this a problem? Sure, someone may be able to generate thousands of real-sounding fake news articles, but it's not like they will also be able to flood the New York Times with these articles. How do you worry you will be exposed to these articles?

It's not me I'm worried about - it's the 50% [1] of people who get their news from social media and "entertainment" news platforms. These people vote, and can get manipulated into performing quite extreme acts. At the moment a lot of people seem to have trouble engaging with reality, and that seems to be caused by relatively small disinformation campaigns and viral rumours. How much worse could it get when there's a…

That fact that you made up that number is extremely funny in this context.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#97

Post by visarga »

anticristi wrote: post #3

I love the initiative, but I'm starting to get scared of what a post-GPT-3 world will look like. We are already struggling to distinguish fake news from real ones, automated customer request replies from genuine replies, etc. How will I know that I have a conversation with a real human in the future? On the other side, the prospect of having an oracle that answers all trivia, fixes spelling and grammar, and allows hu…

> How will I know that I have a conversation with a real human in the future?

This problem should be solved with cryptography, not by banning large neural nets.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#98

Post by rich_sasha »

hooande wrote: post #66

In my experience, the output from GPT-3, DALL-E, et al is similar to what you get from googling the prompt and stitching together snippets from the top results. These transformers are trained on "what was visible to google", which provides the limitation on their utility. I think of the value proposition of GPT-X as "what would you do with a team of hundreds of people who can solve arbitrary problems only by googling…

I think this is exactly right, and indeed this is a lot of the value. "Content-generation" is already a thing, and yes it doesn't need to make much sense. Apparently people who read it don't mind.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#99

Post by visarga »

emphatizer2000 wrote: post #59
dkjaudyeqooe wrote: post #44

Earlier quoted context omitted.

I know! One day it's going to get so bad people are going to have to deploy critical thinking instead of accepting what they read at face value and suffer the indignity of having to think for themselves.

Maybe somebody could create an AI that evaluates the factfulness of articles.

This is possible, especially with human in the loop.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#100

Post by qayxc »

pmontra wrote: post #63
techlatest_net wrote: post #54

Is there any real justification behind this fear of close nature of OpenAI or this is just frustration coming out? We had this debate of closed Vs open source 20 years back and eventually opensource won it because of various reasons. Won't those same reasons apply to this situation of close nature of OpenAI? If so then why are people worried about this? What is differnt this time?

The cost. Closed source and open source developers use the same $300-3,000 laptops / desktops. Everybody can afford them. Training a large model in a reasonable time costs much more. According to https://lambdalabs.com/blog/demystifying-gpt-3/ the cost of training GPT-3 was $4.6 million. Multiply it by the number of trial and errors. Of course we can't expect that something that costs tens or hundreds of millions wil…

This. Plus the increasing amount of intransparent results. Training data is private, so it's impossible to even try to recreate results, validate methods, or find biases/failure cases.
Post reply on HN