Live data from Hacker News

GPT-Neo – Building a GPT-3-sized model, open source and free

eleuther.ai

111–120 of 260 posts

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#111
post #50

Earlier quoted context omitted.

I know! One day it's going to get so bad people are going to have to deploy critical thinking instead of accepting what they read at face value and suffer the indignity of having to think for themselves.

Critical thinking won't help you when the majority (or all) of your sources are tainted and contradictory. At some point, the actual truth just gets swamped.

This is already happening, just with humans networked into social networks that favor quick reshare over deep review

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#112
post #66

In my experience, the output from GPT-3, DALL-E, et al is similar to what you get from googling the prompt and stitching together snippets from the top results. These transformers are trained on "what was visible to google", which provides the limitation on their utility. I think of the value proposition of GPT-X as "what would you do with a team of hundreds of people who can solve arbitrary problems only by googling…

I think this is exactly right, and indeed this is a lot of the value. "Content-generation" is already a thing, and yes it doesn't need to make much sense. Apparently people who read it don't mind.

People don’t read it, search engines do.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#113
post #66

In my experience, the output from GPT-3, DALL-E, et al is similar to what you get from googling the prompt and stitching together snippets from the top results. These transformers are trained on "what was visible to google", which provides the limitation on their utility. I think of the value proposition of GPT-X as "what would you do with a team of hundreds of people who can solve arbitrary problems only by googling…

Did anyone try to ask GPT-3 something about the insurrection? I can't find any free/open text prompts for GPT-3, this is what GPT-2 has to say:

After Trump lost the 2020 election his supporters stormed the Capitol

> They smashed windows, tore apart the Republican Party's headquarters, destroyed a statue of Ronald Reagan and set fire to the Texas statehouse.

> A month later, after the Democrats took control of the Senate and the House, Trump's supporters stormed their own offices. They attacked the offices of California Democratic Senator Kamala Harris and New York Democratic Representative Joe Crowley. They ransacked offices in the Minnesota and Texas statehouses and burned down the Republican Party headquarters in Tennessee. The damage was estimated at

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#114
post #66

In my experience, the output from GPT-3, DALL-E, et al is similar to what you get from googling the prompt and stitching together snippets from the top results. These transformers are trained on "what was visible to google", which provides the limitation on their utility. I think of the value proposition of GPT-X as "what would you do with a team of hundreds of people who can solve arbitrary problems only by googling…

I don't necessarily see the "team of automated googlers" as a fundamental or damning problem with GPT-like approaches. First I think people may have a lot fewer truly original ideas then they are willing to admit. Original thought is sought after and celebrated in arts as a rare commodity. But unlike in arts, where there are almost not constraints, when it comes to science or engineering almost every incremental step is of form Y = Fn(X0,..,Xn) where X0..Xn are widely known and proven to be true. With sufficient logical reasoning and/or experimental data, after numerous peer reviews, we can accept Fn(...) to be a valid transform and Y becomes Xn+1, etc. Before internet or Google one had to go to a library and read books and magazines, or ask other people to find inputs from which new ideas could be synthesized. I think GPT-like stuff is a small step towards automating and speeding up this general synthesis process in the post-Google world.

But if we are looking to replace end-to-end intelligence at scale it's not just about synthesis. We need to also automate the peer review process so that it's bandwidth is matched to increased rate of synthesis. Most good researchers and engineers are able to self-critique their work (and the degree to which they can do that well is really what makes one good IMHO). And then we rely on our colleagues and peers to review our work and form a consensus on its quality. Currently GPT-like systems can easily overwhelm humans with such peer review requests. Even if a model is capable of writing the next great literary work, predicting exactly what happened on Jan 6, or formulating new laws of physics the sheer amount of crap it will produce alongside makes it very unlikely that anyone will notice.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#115

Earlier quoted context omitted.

Photoshop has existed for decades. Is it really that big of a problem for photo news?

The concern is not so much AI generated news, but malicious actors misleading, influencing or scamming people online at scale with realistic conversations. Today we already have large scale automated scams via email and robo call. Less scalable scams like Tinder love catfish scams or Russian/China trolls on reddit are now run by real people, imagine it being automated. If human moderators cannot distinguish these bot…

> imagine not being able to tell if this comment was written by a human or robot

I think neural nets could help finding fake news and factual mistakes. Then it wouldn't matter who wrote it if it is helpful and true.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#116
post #105

Earlier quoted context omitted.

Sure, but that's OpenAI's particular set of circumstances. Generally speaking I struggle to see investors preferring a nebulous non-profit over a for-profit with a clear path to market.

Sure, but we're explicitly talking about OpenAI here.

Of course. It's just that the comment I've been responding to suggested OpenAI going the "open"/non-profit route was to 1) get top researchers and 2) get investment. I was arguing that this doesn't seem to (generally) be a good way to get investment, but I agree with you in that in their case investment just wasn't a consideration at all.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#117
post #106
post #95

Earlier quoted context omitted.

> honestly, not a lot of productive applications come to mind Not so convincing when you enumerate so many applications yourself. > but won't be able to say anything about current events There are variants that use transformer + retrieval, so they got unlimited memory that can be easily extended.

Content farms are hardly a productive application.

You missed the forest for the trees. If you got a tool that can use StackOverflow to solve simple programming tasks, or to generally solve any simple task with Google, then you're sitting on a gold mine.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#118
post #113
post #66

In my experience, the output from GPT-3, DALL-E, et al is similar to what you get from googling the prompt and stitching together snippets from the top results. These transformers are trained on "what was visible to google", which provides the limitation on their utility. I think of the value proposition of GPT-X as "what would you do with a team of hundreds of people who can solve arbitrary problems only by googling…

Did anyone try to ask GPT-3 something about the insurrection? I can't find any free/open text prompts for GPT-3, this is what GPT-2 has to say: After Trump lost the 2020 election his supporters stormed the Capitol > They smashed windows, tore apart the Republican Party's headquarters, destroyed a statue of Ronald Reagan and set fire to the Texas statehouse. > A month later, after the Democrats took control of the Sen…

my god it should be called CNN-2

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#119
post #113
post #66

In my experience, the output from GPT-3, DALL-E, et al is similar to what you get from googling the prompt and stitching together snippets from the top results. These transformers are trained on "what was visible to google", which provides the limitation on their utility. I think of the value proposition of GPT-X as "what would you do with a team of hundreds of people who can solve arbitrary problems only by googling…

Did anyone try to ask GPT-3 something about the insurrection? I can't find any free/open text prompts for GPT-3, this is what GPT-2 has to say: After Trump lost the 2020 election his supporters stormed the Capitol > They smashed windows, tore apart the Republican Party's headquarters, destroyed a statue of Ronald Reagan and set fire to the Texas statehouse. > A month later, after the Democrats took control of the Sen…

GPT-3

> The Trump supporters were armed with guns and knives. They were also carrying torches. They were chanting “Trump 2020” and “Build the Wall.” The Trump supporters were also chanting “Lock her up.” But they were referring to Hillary Clinton.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#120
post #113

Earlier quoted context omitted.

Did anyone try to ask GPT-3 something about the insurrection? I can't find any free/open text prompts for GPT-3, this is what GPT-2 has to say: After Trump lost the 2020 election his supporters stormed the Capitol > They smashed windows, tore apart the Republican Party's headquarters, destroyed a statue of Ronald Reagan and set fire to the Texas statehouse. > A month later, after the Democrats took control of the Sen…

GPT-3 > The Trump supporters were armed with guns and knives. They were also carrying torches. They were chanting “Trump 2020” and “Build the Wall.” The Trump supporters were also chanting “Lock her up.” But they were referring to Hillary Clinton.

This is hilarious, more please :)
Post reply on HN