Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI's GPT-3 may be the biggest thing since Bitcoin

maraoz.com

Post reply on HN
Print view View original

61–70 of 554 posts

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#61

Post by threeseed »

rvz wrote: post #2

> I further predict that this will spark a creative gold rush among talented amateurs to train similar models and adapt them to a variety of purposes, including: mock news, “researched journalism”, advertising, politics, and propaganda. The first mention of 'Elon Musk' (who left the board) and this sentence alone gave me the tip-off that GPT-3 had generated that (and the whole blog) and it's following prediction make…

> no-one can train GPT-3 in any acceptable time except for those with access to large GPU/ASIC compute power

That's not true.

a) Premium AWS customers, for example, can request to have instance limits removed which then gives you access to all of the AWS GPU-enabled instances available worldwide.

b) People ramble on about GPUs but Intel DLBoost/AVX-512 enabled CPUs can get you comparable performance to a medium end GPU in many situations. That then opens the door to training across all of the cloud and VPS providers.

Money is the limiting factor here not available compute resources.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#62

Post by cli »

qeternity wrote: post #19

Oh man. I don’t know where to begin. I’ll just say that this analysis is predicated on someone who thinks Bitcoin has proven anything in the real world. We’re way past BTC being a subculture experiment. Everyone knows about it. And still nobody (in the statistical sense) uses it.

Are you referring only to the title?

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#63

Post by Alex3917 »

meredydd wrote: post #28

I am deeply enjoying this comment thread - it's a bit of a Barium Meal [0] for determining how many people read (a) the headline, (b) the first paragraph, or (c) the whole thing before jumping straight into the compose box. Having read to the bottom, the quality of text generation there absolutely blew me away. GPT-2 texts have a somewhat disconnected quality - "it only makes sense if you're not really paying attenti…

> Even on re-reading more closely, it doesn't feel like the world's best writing, but I don't notice major loss of coherence until the last couple of paragraphs.

I guessed it was fake before getting to the end, not from the content, but from the fact that all the sentences are roughly the same length and follow the same basic grammatical patterns. Real people purposely mix up their sentence structure in order to keep the readers engaged, whereas this wasn't doing that at all. Still very impressive though; if not for the fact that the post was about computer generated content I probably wouldn't have noticed.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#64

Post by m3kw9 »

meredydd wrote: post #28

I am deeply enjoying this comment thread - it's a bit of a Barium Meal [0] for determining how many people read (a) the headline, (b) the first paragraph, or (c) the whole thing before jumping straight into the compose box. Having read to the bottom, the quality of text generation there absolutely blew me away. GPT-2 texts have a somewhat disconnected quality - "it only makes sense if you're not really paying attenti…

Is exactly the issue. You still need humans to check it before releasing an output. It can only be what the author says “Bitcoin” level of implication if it can get things probably needing at least “99%” quality and correctness.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#65

Post by h0p3 »

jcahill wrote: post #53
meredydd wrote: post #28

I am deeply enjoying this comment thread - it's a bit of a Barium Meal [0] for determining how many people read (a) the headline, (b) the first paragraph, or (c) the whole thing before jumping straight into the compose box. Having read to the bottom, the quality of text generation there absolutely blew me away. GPT-2 texts have a somewhat disconnected quality - "it only makes sense if you're not really paying attenti…

GPT-3 is a neat party trick. But the things that'll be done with web archives* in the next 20y will make it look like the PDP-8. ~love, a web archivist * GPT-3 is trained on one

No pressure: feel free to ignore me, please. Would you mind elaborating? I'm interested in what you have to say (and, of course, feel free to say it privately if you prefer). I would like to even hear your dreams, wild speculations, or gut feelings about the matter.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#66

Post by bouncycastle »

The irony is bitcoin didn't disrupt anything. Most of it is propped up by memes & hype, that made a few people rich. While it was impressive at the beginning and made many news stories, the technology itself is unsustainable, unscalable, with big usability problems. Disappointing to say the least. (Perhaps that's what the author of the article was subtly implying?)

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#67

Post by Kinrany »

teruakohatu wrote: post #41
nbardy wrote: post #9

Earlier quoted context omitted.

One of the biggest issues is with cherry-picking. Generative ML results benefit greatly from humans sampling the best results. They are capable of producing astonishing results but don’t do this consistently this has a huge impact on any effort to productize. For example I’ve seen quite a few examples of text->design, text->code, with GPT-3 you could build a demo in a day, but the product will probably be useless if…

I don't know about GPT-3 but playing around with GPT-2 I often got the impression that it was regurgitating learned knowledge (reddit comments) rather than actually coming up with something novel. With so many weights, it practically encodes a massive Internet text database.

Having a better alternative to search engines would be great.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#68

Post by Imnimo »

nullc wrote: post #55
Imnimo wrote: post #48

I'm surprised that GPT-3 understands how to write coherently about GPT-3. Common Crawl, the bulk of GPT-3's training data, was collected even before GPT-2 was published. Maybe the Wikipedia dump they used was recent enough to contain a few references to GPT-2, but that's it. So if GPT-3 never saw any discussion of GPT-3, and probably almost no discussion of GPT-2, in its training data, how does it know what they are?…

I mean, you could write coherently about GPT3 without ever hearing about it before-- right? You could absolutely write coherently about it with a bit of priming text, ... I mean, that is exactly what you're doing right now. :) Unless you work for open-ai, you've never actually seen GPT3... for all you know for sure it could be some amazing mechanical turk thing postprocessing GPT2 output :)

I suppose I could have guessed that GPT-2 is a plausible name for the predecessor of GPT-3, and the other predicted predecessor name (PTB) is not really correct. But I'm still surprised that it can hit a believable tone about its own capabilities. I don't think I could have done that if you had asked me before GPT-2 was published (which is essentially what we're doing if GPT-3's training data predates GPT-2). I think I would have under-estimated and it would be immediately clear to a reader familiar with GPT-3 that I didn't actually know what it could or could not do.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#69

Post by garbagetime »

briga wrote: post #45
graham_paul wrote: post #7

Earlier quoted context omitted.

> no-one can train GPT-3 in any acceptable time except for those with access to large GPU/ASIC compute power (OpenAI, Microsoft, Google, NVIDIA, etc.) Any state actor has access to large compute power

But it might be hard to attract talented AI engineers to live in, say, North Korea

Koreans can become talented AI engineers.

The DPRK has a history of simply abducting people with the talents that the DPRK requires. A plausible (although unlikely) scenario is that the DPRK abducts a handful of AI experts and forces them to train Koreans in AI.

Alternatives are for Koreans to learn abroad or for Koreans to learn online.

Post reply on HN