Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-Neo – Building a GPT-3-sized model, open source and free

eleuther.ai

Post reply on HN
Print view View original

151–160 of 260 posts

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#151

Post by mistermann »

jokethrowaway wrote: post #137
chimprich wrote: post #93

Earlier quoted context omitted.

It's not me I'm worried about - it's the 50% [1] of people who get their news from social media and "entertainment" news platforms. These people vote, and can get manipulated into performing quite extreme acts. At the moment a lot of people seem to have trouble engaging with reality, and that seems to be caused by relatively small disinformation campaigns and viral rumours. How much worse could it get when there's a…

The majority of "fake news" are factual news described from a partial point of view and with a political spin. Even fact checkers are not immune to this and brand other news as true or false not based on facts but based on the political spin they favour. Fake news is a vastly overstated problem. Thanks to internet, we now have a wider breadth of political news and opinions and it's easy to label everything-but-your-s…

> There are a few patently false lies on the internet which are taken as examples of fake news - but they have very few supporters.

Very true. What's interesting though is how many supporters there of the idea that extremely large quantities of people 100% "hook, line, and sinker" buy into such fake news stories - based, ironically, on fake-news-like articles assuring us (with specious evidence, if any) that this is the true state of reality.

The world is amazingly paradoxical if you look at it from the proper abstract angle.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#152

Post by orm »

hooande wrote: post #66

In my experience, the output from GPT-3, DALL-E, et al is similar to what you get from googling the prompt and stitching together snippets from the top results. These transformers are trained on "what was visible to google", which provides the limitation on their utility. I think of the value proposition of GPT-X as "what would you do with a team of hundreds of people who can solve arbitrary problems only by googling…

GPT-3 is trained on text prediction, and there's been a lot of commentary about the generation aspect, but some of the applications that excite me most are not necessarily about the generation of text, instead, GPT-3 (and also other language models) create very useful vector representations of natural language as a side effect that can then be used for other tasks with much less data, or with too much extra data. Using the text prediction task as a way to supervise learning this representation without having to create an expensive labelled dataset is very helpful, and not just to language tasks. See for example the CLIP work that came out recently for image classification, using GPT-3 and captions to supervise training. There is other work referred to in that blog post that also exploits captions or descriptions in natural language to help understand images better. More speculatively, being able to use natural language to supervise or give feedback to automated systems that have little to do with NLP seems very very useful.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#153

Post by eeZah7Ux »

Closi wrote: post #126
pessimizer wrote: post #110

Earlier quoted context omitted.

I don't know where people got the idea that companies can be "sincere." Sincerity is faithfully communicating your mental state to others. A company's mental state can change on a dime based on the decisionmaking of people who rely on the company for the degree of profit it generates. Any analog to sincerity that you think you see can probably be eliminated by firing one person after an exceptionally bad quarter (or…

Sincere to me just means that you are being truthful, or not trying to be deceptive. And I think companies can be sincere - because companies are really just groups of people and assets when you get down to the nuts and bolts of it.

> companies can be sincere

"sincere", "honest", "hypocritical" usually refers to a long-term pattern. Being able to be sincere from time to time is besides the point.

> companies are really just groups of people

...with profit as their first priority.

For-profit companies "can be sincere" only as long as it's the most profitable strategy.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#154

Post by IndySun »

darkwater wrote: post #60
IndySun wrote: post #53

Earlier quoted context omitted.

"one of the things that should absolutely be taught in school but isn't." Namely, critical thinking?

I don't think "critical thinking" is the point here. Because first you need to know that such modifications CAN be done. And not everybody knows what can be retouched with PS or programs. So yeah, if you see some super-model on a magazine cover, and you don't know PS can edit photos easily, it would be not that immediate to think "hey maybe that's not real!". As an extreme example: would you ever checked 20 years ago…

I think I made my point badly because I also agree.

I am lamenting that teenagers were, in this day and age, surprised at what can be done with Photoshop. And that let loose on the appropriate software were surprised at what can be altered and how easily.

My point is suggesting this may be so because people have not been taught how to think for themselves and accept things (in this case female images) 'as is', without a hint of curiosity. It is also a problem but at the other end of the stick, with many young people I work with considering Wikipedia to be 100% full of misinformation and fake news.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#155

Post by masswerk »

throwaway6e8f wrote: post #112
rich_sasha wrote: post #98

Earlier quoted context omitted.

I think this is exactly right, and indeed this is a lot of the value. "Content-generation" is already a thing, and yes it doesn't need to make much sense. Apparently people who read it don't mind.

People don’t read it, search engines do.

BTW, we should mandatorily tag generated content for search engines in order to exclude it from future training sets.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#156

Post by chimprich »

profunctor wrote: post #96
chimprich wrote: post #93

Earlier quoted context omitted.

It's not me I'm worried about - it's the 50% [1] of people who get their news from social media and "entertainment" news platforms. These people vote, and can get manipulated into performing quite extreme acts. At the moment a lot of people seem to have trouble engaging with reality, and that seems to be caused by relatively small disinformation campaigns and viral rumours. How much worse could it get when there's a…

That fact that you made up that number is extremely funny in this context.

I don't think so - I was aware that it was a made-up number, and highlighted the fact that it was. It's the lack of awareness of what is backed up by data that is the problem I think.

Or am I missing your point?

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#157

Post by londons_explore »

newswasboring wrote: post #65
londons_explore wrote: post #45

Earlier quoted context omitted.

It does. For most models, your home broadband would be far too slow though.

Is it because they will have to communicate back errors during training? I forgot that training these models is more of a global task than proteins folding. In that sense this is less parallelizable over the internet.

Yes, and also activations if your GPU is too small to fit the whole model. The minimum useful bandwidth for that stuff is a few gigabits...

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#158

Post by chimprich »

jokethrowaway wrote: post #137
chimprich wrote: post #93

Earlier quoted context omitted.

It's not me I'm worried about - it's the 50% [1] of people who get their news from social media and "entertainment" news platforms. These people vote, and can get manipulated into performing quite extreme acts. At the moment a lot of people seem to have trouble engaging with reality, and that seems to be caused by relatively small disinformation campaigns and viral rumours. How much worse could it get when there's a…

The majority of "fake news" are factual news described from a partial point of view and with a political spin. Even fact checkers are not immune to this and brand other news as true or false not based on facts but based on the political spin they favour. Fake news is a vastly overstated problem. Thanks to internet, we now have a wider breadth of political news and opinions and it's easy to label everything-but-your-s…

> Even fact checkers are not immune to this and brand other news as true or false not based on facts but based on the political spin they favour.

Could you give an example?

> There are a few patently false lies on the internet which are taken as examples of fake news - but they have very few supporters.

How many do you consider "few"?

I can go to my local news site and read a story about the novel coronavirus and the majority of comments below the article are stating objectively false facts.

"It's just a flu" "Hospitals are empty" "The survival rate is 99.9%" "Vaccines alter your DNA"

...and so on.

There is the conspiracy theory or cult called QAnon, which "includes in its belief system that President Trump is waging a secret war against elite Satan-worshipping paedophiles in government, business and the media."

One QAnon Gab group has more than 165,000 users. I don't think these are small numbers.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#160

Post by ccostes »

Sambdala wrote: post #73
Erlich_Bachman wrote: post #4

It's a shame that it has turned out to be necessary to externally re-make and re-train a model that has come out of company called `OPEN`AI. Wasn't one of the founding principles of it that all of the research would be available to the public? Isn't that the premise on which the initial funding was secured? Best of luck to Eleuther.

Wild-Ass Guess (Ass-Guess) incoming: OpenAI was built to influence the eventual value chain of AI in directions that would give the funding parties more confidence that their AI bets would pay off. This value chain basically being one revolving around AI as substituting predictions and human judgement in a business process, much like cloud can be (oversimply) modeled as moving Capex to Opex in IT procurement. They sa…

I think calling this a "wild-ass guess" undersells it a bit (either that or we have very different definitions of a WAG).Very well though-through and compelling case.

My biggest question is whether composable models are indeed the general case, which you say they confirmed as evidenced by the shift away from non-profit. It's certainly true for some domains, but I wonder if it's universal enough to enable the ecosystem you describe.

Post reply on HN