Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-Neo – Building a GPT-3-sized model, open source and free

eleuther.ai

Post reply on HN
Print view View original

141–150 of 260 posts

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#141

Post by mrfusion »

Filligree wrote: post #140
mrfusion wrote: post #121

It still baffles me that GPT turned out to be more than a glorified markov chain text generator. It seems we’ve actually made it create a model of the world to some degree. And we kind of just stumbled on the design by throwing massive data and neural networks together?

It turns out that brute-force works, and the scaling curve is still not bending. I doubt we'll ever see a GPT-4, because there are known improvements they could make besides just upsizing it further, but that's besides the point. If that curve doesn't bend soon then a 10x larger network would be human-level in many ways. (Well, that is to say. It's actually bending. Upwards.)

So fascinating. I’d love to understand why it’s working so well. I guess no one knows.

Wouldn’t gpt4 just be more data and more parameters?

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#142

Post by yowlingcat »

Lucasoato wrote: post #88
hooande wrote: post #66

In my experience, the output from GPT-3, DALL-E, et al is similar to what you get from googling the prompt and stitching together snippets from the top results. These transformers are trained on "what was visible to google", which provides the limitation on their utility. I think of the value proposition of GPT-X as "what would you do with a team of hundreds of people who can solve arbitrary problems only by googling…

> I think of the value proposition of GPT-X as "what would you do with a team of hundreds of people who can solve arbitrary problems only by googling them?". And honestly, not a lot of productive applications come to mind. Damn, this could replace so many programmers, we're doomed!

Curses. We've been found out.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#143

Post by eeZah7Ux »

dvfjsdhgfv wrote: post #75
wraptile wrote: post #19

Earlier quoted context omitted.

OpenAI turning out to be a total bait and switch. Especially true when your co-founder is actively calling you out on it[1] Remember kids: if it's not a non-profit organization it is a _for_ profit one! It was silly to expect anything else: > In 2019, OpenAI transitioned from non-profit to for-profit. The company distributed equity to its employees and partnered with Microsoft Corporation, who announced an investment…

It will be interesting to see the attitude of Microsoft towards this project in the light of their "Microsoft loves open source" propaganda.

Like many other companies, Microsoft loves unpaid labor.

Free Software is about giving freedom and security all the way to the end users - rather than SaaS providers.

If you remove this goal and only focus on open source as a development methodology you end up with something very similar to volunteering for free for some large corporation.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#144

Post by schaefer »

hooande wrote: post #66

In my experience, the output from GPT-3, DALL-E, et al is similar to what you get from googling the prompt and stitching together snippets from the top results. These transformers are trained on "what was visible to google", which provides the limitation on their utility. I think of the value proposition of GPT-X as "what would you do with a team of hundreds of people who can solve arbitrary problems only by googling…

I feel your stance [1] is demonstrably false in two challenges.

1) Please play a winning game of Go against Alpha Zero, just by googling the topic.

2) Next please explain how Alpha Zero’s game’s could forever change Go opening theory[2], without any genuine creativity.

[1] that “the output from GPT-3, DALL-E, et al is similar to what you get from googling the prompt and stitching together snippets from the top results.”

[2]”Rethinking Opening Strategy: AlphaGo's Impact on Pro Play” by Yuan Zhou

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#145

Post by mistermann »

hooande wrote: post #66

In my experience, the output from GPT-3, DALL-E, et al is similar to what you get from googling the prompt and stitching together snippets from the top results. These transformers are trained on "what was visible to google", which provides the limitation on their utility. I think of the value proposition of GPT-X as "what would you do with a team of hundreds of people who can solve arbitrary problems only by googling…

> I think of the value proposition of GPT-X as "what would you do with a team of hundreds of people who can solve arbitrary problems only by googling them?". And honestly, not a lot of productive applications come to mind.

If I was Xi Jinping, I would use it to generate arbitrary suggestions for consideration by my advisory team, as I develop my ongoing plan for managing The Matrix.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#146

Post by ravi-delia »

schaefer wrote: post #144
hooande wrote: post #66

In my experience, the output from GPT-3, DALL-E, et al is similar to what you get from googling the prompt and stitching together snippets from the top results. These transformers are trained on "what was visible to google", which provides the limitation on their utility. I think of the value proposition of GPT-X as "what would you do with a team of hundreds of people who can solve arbitrary problems only by googling…

I feel your stance [1] is demonstrably false in two challenges. 1) Please play a winning game of Go against Alpha Zero, just by googling the topic. 2) Next please explain how Alpha Zero’s game’s could forever change Go opening theory[2], without any genuine creativity. [1] that “the output from GPT-3, DALL-E, et al is similar to what you get from googling the prompt and stitching together snippets from the top result…

Op was clearly not talking about Alpha Zero, a different technology made by different people for a different purpose. Instead, they were noting that despite displaying some truly excellent world modeling, GPT-3 is trained on data that encourages it to vomit up rehashes. It's very possible that the next generation will overcome this and wind up completely holding together long-run concepts and recursion, at least if scaling parameters keeps working, but for now it is a real limitation.

GPT-3 writes like a sleepy college student with 30 minutes before the due date; with shockingly complete grasp of language, but perhaps not complete understanding of content. That's not just an analogy, I am a sleepy college student. When I write an essay without thinking too hard it displays exactly the errors that GPT-3 makes.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#147

Post by tbenst »

schaefer wrote: post #144
hooande wrote: post #66

In my experience, the output from GPT-3, DALL-E, et al is similar to what you get from googling the prompt and stitching together snippets from the top results. These transformers are trained on "what was visible to google", which provides the limitation on their utility. I think of the value proposition of GPT-X as "what would you do with a team of hundreds of people who can solve arbitrary problems only by googling…

I feel your stance [1] is demonstrably false in two challenges. 1) Please play a winning game of Go against Alpha Zero, just by googling the topic. 2) Next please explain how Alpha Zero’s game’s could forever change Go opening theory[2], without any genuine creativity. [1] that “the output from GPT-3, DALL-E, et al is similar to what you get from googling the prompt and stitching together snippets from the top result…

GPT-3 can’t play Go.

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#148

Post by mistermann »

Drakim wrote: post #30
taneq wrote: post #14

Earlier quoted context omitted.

The fake news thing is a real problem (and may become worse under GPT3 but certainly exists already). As for the others - to quote Westworld, "if you can't tell the difference, does it really matter?"

What scares me personally is the idea that I might be floating in a sea of uncanny valley content. Content that's 98% human-like, but then that 2% sticks out like a nail and snaps me out of it. Sure, I might not be able to tell the difference the majority of the time, but when I can tell the difference it's gonna bother me a lot.

Do you not already have this feeling on a fairly regular basis? (Serious question)

Re: GPT-Neo – Building a GPT-3-sized model, open source and free

#149

Post by PaulHoule »

tigerBL00D wrote: post #114
hooande wrote: post #66

In my experience, the output from GPT-3, DALL-E, et al is similar to what you get from googling the prompt and stitching together snippets from the top results. These transformers are trained on "what was visible to google", which provides the limitation on their utility. I think of the value proposition of GPT-X as "what would you do with a team of hundreds of people who can solve arbitrary problems only by googling…

I don't necessarily see the "team of automated googlers" as a fundamental or damning problem with GPT-like approaches. First I think people may have a lot fewer truly original ideas then they are willing to admit. Original thought is sought after and celebrated in arts as a rare commodity. But unlike in arts, where there are almost not constraints, when it comes to science or engineering almost every incremental step…

Art (music, literature, ...) involves satisfaction of constraints. For instance you need to tune your guitar like the rest of the band, write 800 words like the editor told you, tell a story with beginning, middle, and end and hopefully not use the cheap red pigments that were responsible for so many white, blue, and gray flags I saw in December 2001.
Post reply on HN