Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI's GPT-3 may be the biggest thing since Bitcoin

maraoz.com

Post reply on HN
Print view View original

141–150 of 554 posts

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#141

Post by httpsterio »

After reading the AI generated sections, I have to say that I'm mostly quite impressed. As I lack the context as I am not actively following the ML and procedurally generated text scenes for years, it was still just markov chains back then, I can't say for sure how accurate the produced text was.

It's scary though. Many commenters are only discussion about the business opportunities and path to profitability, but if we step back and stop to think for a while what GPT-3 level tech enables us to do. Robocalls, spam articles and bot generated forum posts are already all too common and while not totally impossible to distinguish, I fear that this level of text generation will only make matters far worse.

I'm personally a moderator on a 300k user subreddit which is basically 90% text and very little links, no images or videos, and we are already facing challenges with distinguishing botted marketing campaigns. I fear that in the future it will be even harder to distinguish if you're actually talking with another human being on example a support chat or customer service. The future looks like a Blade Runner esque dystopian landscape of diminishing individuality.

I'm doubtful that any attempts at regulating or containing the possible issues would be possible as the barrier for entry has never been lower. Anyone with a decent gaming PC these days can start training their models in mere hours or days.

Am I being paranoid? Maybe. Like said, I'm out of the loop and I would definitely love to hear some halming words quenching my fears.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#142

Post by yomly »

ypcx wrote: post #124
jcahill wrote: post #53

Earlier quoted context omitted.

GPT-3 is a neat party trick. But the things that'll be done with web archives* in the next 20y will make it look like the PDP-8. ~love, a web archivist * GPT-3 is trained on one

The transformer model as presented in GPT-3 may be a few tweaks away from a human-acceptable reasoning, at which point we may realize that human brain is just a neat party trick as well. This may come difficult for some people to internalize, especially those who understand the technology in depth. Because it means that the medium of our reality is the consciousness.

I don't fully understand what you're getting at here...

Basically the brain and "consciousness" isn't as fancy as we think?

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#143

Post by redm »

I'm catching up on the GPT-* news, but its pretty interesting.

"GPT-3 is the latest in a series of text-generating neural networks. The name GPT stands for Generative Pretrained Transformer, referencing a 2017 Google innovation called a Transformer which can figure out the likelihood that a particular word will appear with surrounding words. Fed with a few sentences, such as the beginning of a news story, the GPT pre-trained language model can generate convincingly accurate continuations, even including the formulation of fabricated quotes."

It sounds a lot like a fancy version of PageRank for words. The results are impressive though. Like Grammerly + PageRank.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#144

Post by bkanber »

httpsterio wrote: post #141

After reading the AI generated sections, I have to say that I'm mostly quite impressed. As I lack the context as I am not actively following the ML and procedurally generated text scenes for years, it was still just markov chains back then, I can't say for sure how accurate the produced text was. It's scary though. Many commenters are only discussion about the business opportunities and path to profitability, but if…

It'll just be yet another technical advancement that we have to adapt to. And we will.

What gives me comfort is that there's still an operator with a motive. It's funny, as I was reading your comment I half expected it to end with "this was generated by GPT-3". And it wouldn't have mattered, because you-the-person still had a sentiment or a message you wanted to communicate, and then you communicated it, whether or not you actually wrote the message.

It _would_ be better if support chat understood me better and could communicate with me better. Because again, the motive is understood and aligns with mine. I don't necessarily need a human to do the typing. Typing is a lot of work.

In your example, marketing campaigns will continue to be marketing campaigns. I don't worry that the bots will flood your sub more than they already are; if they flood it too much, they destroy any value they hoped to leverage in the first place. The only difference will be that the language in between the spammy links will be more readable.

If someone on a, eg, adult chat forum is whispering sweet nothings at you, again I don't care if an AI wrote the prose; someone still decided to whisper sweet nothings at you. They just hired GPT-3 to do the writing instead of doing a sloppy job themselves.

I think the real problems start when the AI decides to initiate the action. When an AI, not explicitly instructed by a human operator, decides to launch a marketing campaign. That's the dystopia I'm worried about.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#145

Post by enjoylife »

canjobear wrote: post #122
abernard1 wrote: post #96

When I read comments like this--and yes I read the article and understand it was generated by an algorithm--I can't help but think the next AI winter is around the corner. This does not impress me in the slightest. Taking billions and billions of input corpora and making some of them _sound like_ something a human would say is not impressive. Even if it's at a high school vocabulary level. It may have underlying corr…

I think you are underestimating what an advance these models are over previous NLP models in terms of quality. Before GPT-2 we didn't even have models that could reliably generate grammatical sentences. Now we have things that generate coherent (if not beautiful) paragraphs. It seems easy in retrospect, but some of the smartest people around have been working on this for decades.

I marvel at the jump in BLEU or other measures but I'll second the sentiment, it alone is not showing we are making leaps toward what we need. Yes its a large gradient step minimizing our error, but is it really in the right direction? However I will admit GPT-3 being directed by some yet to be invented causal or counterfactual inference model might be the something which defies my expectations.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#146

Post by wcoenen »

m3kw9 wrote: post #106
meredydd wrote: post #28

I am deeply enjoying this comment thread - it's a bit of a Barium Meal [0] for determining how many people read (a) the headline, (b) the first paragraph, or (c) the whole thing before jumping straight into the compose box. Having read to the bottom, the quality of text generation there absolutely blew me away. GPT-2 texts have a somewhat disconnected quality - "it only makes sense if you're not really paying attenti…

Problem was I lost interest half way because it lost my interest after the 2nd paragraph. For those that say it was good till the last few is really pretending to understand what it said. It really did not made much sense.

It's ironic that you're critiquing the algorithm for not making sense, while contradicting yourself in your first sentence. Did you lose interest "half way", or "after the second paragraph"? It can't be both.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#147

Post by cs702 »

thewarrior wrote: post #133
meredydd wrote: post #28

I am deeply enjoying this comment thread - it's a bit of a Barium Meal [0] for determining how many people read (a) the headline, (b) the first paragraph, or (c) the whole thing before jumping straight into the compose box. Having read to the bottom, the quality of text generation there absolutely blew me away. GPT-2 texts have a somewhat disconnected quality - "it only makes sense if you're not really paying attenti…

I've started working on a version of GPT-2 which generates English text. The purpose of this is to improve its ability to predict the next character in a text, by having it learn 'grammatical rules' for English. It already works well for predicting the next character when it has seen only a small amount of text, but becomes less accurate as the amount of training text increases. I have managed to improve this by havi…

The thing that kills me is that to the vast majority of human beings the nonsensical technobabble above is probably indistinguishable from real, honest, logically consistent technobabble.[a]

Soon enough, someone will replicate the Sokal hoax[b] with GPT-3 or another state-of-the-art language-generation model. It's not hard to imagine GPT-3 writing a fake paper that gets published in certain academic journals in the social sciences.

[a] https://en.wikipedia.org/wiki/Technobabble

[b] https://en.wikipedia.org/wiki/Sokal_affair -- here's a copy of Sokal's hoax paper, "Transgressing the Boundaries: Towards a Transformative Hermeneutics of Quantum Gravity:" https://physics.nyu.edu/faculty/sokal/transgress_v2/transgre...

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#148

Post by shpongled »

nullc wrote: post #84

I've been using GPT2 as a writing aid for a little while (the 1558M model is a lot better than the smaller ones). Some of my recently highly upvoted HN comments were substantially written by GPT-2. There is quite a bit of art that goes into crafting prompts that produce useful output. There are a few things about GPT2 that I didn't like (for instance, the number of "quoted text" blocks) but I have found that if I put…

Was this comment written by GPT-2? Is it jealous of GPT-3?

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#149

Post by exolymph »

abernard1 wrote: post #96

When I read comments like this--and yes I read the article and understand it was generated by an algorithm--I can't help but think the next AI winter is around the corner. This does not impress me in the slightest. Taking billions and billions of input corpora and making some of them _sound like_ something a human would say is not impressive. Even if it's at a high school vocabulary level. It may have underlying corr…

> This does not impress me in the slightest.

A computer that is actually fluent in English — as in, understands the language and can use it context-appropriately — should blow your entire mind.

Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin

#150

Post by quotemstr »

oehtXRwMkIs wrote: post #85
quotemstr wrote: post #50

Earlier quoted context omitted.

And how do we indicate that any of these processes has occurred except through the medium of language? A sufficiently good text predictor is a sentient mind. I don't believe that there's any experience of consciousness distinct from the use of language.

For most of human history text has not existed. I think you are conflating language with written language which is quite common in the post-Gutenberg age. I think sentience like most things is a spectrum, so I'm not really sure what you mean by sentient, but I would argue that for most people the bar for sentience is much higher than text prediction. The Chinese room is only one aspect of our minds, and we don't even…

I have to disagree with this perspective and reiterate my original position: language is sentience.

And to be fair, reasonable people stake out positions on both sides of this debate: I'm not claiming that the alternative proposition is somehow unreasonable. It's a legitimate subject of scholarly disagreement.

Nevertheless, I'm still firm on language. Why? Because all complexity is ultimately about symbolic manipulation of terms representing the process of manipulation itself. ("Godel, Escher, Bach" is a fantastic exploration of this concept.) How can you manipulate concepts without assigning terms to their parts? That's what language is.

The question I like to ask is this: are there any ideas that you cannot express using language? No? Then how is thought distinct from language?

Yes, people (myself included) experience a "tip of the tongue" experience where you feel like you have an idea you can't just yet express. But maybe this experience is what reason feels like. Why should idea formation take only one "clock cycle" in the brain? Why should we be unaware of the process?

I think this feeling of having an idea yet being unable to formulate it is just the neural equivalent of a CPU pipeline stall. It's not evidence that we can have ideas without language: it's evidence that ideas sometime take a little while to gel.

Post reply on HN