Very impressive! I read this while watching the California sun setting on a lonely (but cheerful!) Saturday night. I laughed out loud when I got to the reveal and clearly disturbed my cats. This is perhaps the first time I’ve understood the power of AI. And I’ve read multiple books on the potential and watched an inane amount of videos. Many thanks @maraoz for putting this together.
OpenAI's GPT-3 may be the biggest thing since Bitcoin
161–170 of 554 posts
Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin
#162Earlier quoted context omitted.
I've started working on a version of GPT-2 which generates English text. The purpose of this is to improve its ability to predict the next character in a text, by having it learn 'grammatical rules' for English. It already works well for predicting the next character when it has seen only a small amount of text, but becomes less accurate as the amount of training text increases. I have managed to improve this by havi…
The thing that kills me is that to the vast majority of human beings the nonsensical technobabble above is probably indistinguishable from real, honest, logically consistent technobabble.[a] Soon enough, someone will replicate the Sokal hoax[b] with GPT-3 or another state-of-the-art language-generation model. It's not hard to imagine GPT-3 writing a fake paper that gets published in certain academic journals in the s…
In a similar way to how image detection networks appear to key largely on visual textures, GPT-3 seems to key on jargon, tone and pacing, the texture of speech within specific milieus.
Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin
#163When I read comments like this--and yes I read the article and understand it was generated by an algorithm--I can't help but think the next AI winter is around the corner. This does not impress me in the slightest. Taking billions and billions of input corpora and making some of them _sound like_ something a human would say is not impressive. Even if it's at a high school vocabulary level. It may have underlying corr…
> I'll be impressed the day I can see a program that can 1) only rely on its own limited experiential inputs Hasn't the typical human taken in orders of magnitude more data than this example? And the data has been of both direct sensory experience and texts from other people as well.
> Hasn't the typical human taken in orders of magnitude more data than this example?
Have you read GPT-3s 175 billion parameters (words, sentences, papers, I don't care) of anything? Do you know all the words used in that corpus? Nobody has or does.
A child of a small age can listen to a very small set of things and not just come up with words to communicate with mama and papa what they learned, but they can reuse it. And this I think is key, because the language part of that is at least partially secondarily. The little kid understands what they're talking about even if they have a hard time communicating it to an adult. The fact they take creative leaps to use their extremely limited vocabulary to communicate their knowledge is amazing.
Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin
#164Earlier quoted context omitted.
I've started working on a version of GPT-2 which generates English text. The purpose of this is to improve its ability to predict the next character in a text, by having it learn 'grammatical rules' for English. It already works well for predicting the next character when it has seen only a small amount of text, but becomes less accurate as the amount of training text increases. I have managed to improve this by havi…
The thing that kills me is that to the vast majority of human beings the nonsensical technobabble above is probably indistinguishable from real, honest, logically consistent technobabble.[a] Soon enough, someone will replicate the Sokal hoax[b] with GPT-3 or another state-of-the-art language-generation model. It's not hard to imagine GPT-3 writing a fake paper that gets published in certain academic journals in the s…
It's not hard to imagine GPT-3 writing a fake paper that gets published in certain academic journals in the social sciences. And then, it'll be all over for us. We won't have any more funding and our jobs will disappear. I can already hear the protests: "But we're not just scientists! We're also philosophers!" Well, yes and no. Philosophers are supposed to think about things philosophically, but they don't actually do anything about them; they're just entertainers. Scientists do something about them. They make things happen. And when those things happen, people take notice. If science and technology have a weakness, it's that they work too well. This was probably a strength at one point, but not anymore. In the not-too-distant future, there probably won't be any more philosophy professors; there will just be philosophers. But only in the same sense that there are lions and mushrooms.
This comment was also written by GPT-3.
Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin
#165Earlier quoted context omitted.
Problem was I lost interest half way because it lost my interest after the 2nd paragraph. For those that say it was good till the last few is really pretending to understand what it said. It really did not made much sense.
It's ironic that you're critiquing the algorithm for not making sense, while contradicting yourself in your first sentence. Did you lose interest "half way", or "after the second paragraph"? It can't be both.
Wouldn't "after the second paragraph" be "half-way" for a four paragraph piece? :D
But you are right, it can't be both in the context of this article :)
Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin
#166Earlier quoted context omitted.
> Maybe. Right now this reads like a glorified shopping list. It's coherent, but actually sounding human also requires a theory of mind. I'm going to call this goalpost shifting. This article is better writing than some % of humans, theory of mind or otherwise. The AI has comfortably surpassed Timecube-level writing and is entering the pool of passes-for-a-human. 'Sounds human' is a spectrum that starts with the ment…
> I'm going to call this goalpost shifting. That's completely fair. On the other hand, without a theory of mind it can't really educate or inspire people, the only thing it can do is maybe trick them about the authorship of something. But once people learn the techniques for identifying this kind of writing, it can't even do that anymore. To me this is like the front end of something, but it still needs a back end. D…
On the other hand, without a theory of mind it can't really educate or inspire people
I wouldn't agree with that, either. How often have we heard of someone gaining useful insights by considering ideas that were misapplied or just plain wrong? Entire branches of physics have evolved that way. As far as successful presidential campaigns are concerned... well, let's not even go there.
If there's such a thing as a 'theory of mind', it applies to the reader, not the writer.
Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin
#167Earlier quoted context omitted.
For most of human history text has not existed. I think you are conflating language with written language which is quite common in the post-Gutenberg age. I think sentience like most things is a spectrum, so I'm not really sure what you mean by sentient, but I would argue that for most people the bar for sentience is much higher than text prediction. The Chinese room is only one aspect of our minds, and we don't even…
I have to disagree with this perspective and reiterate my original position: language is sentience. And to be fair, reasonable people stake out positions on both sides of this debate: I'm not claiming that the alternative proposition is somehow unreasonable. It's a legitimate subject of scholarly disagreement. Nevertheless, I'm still firm on language. Why? Because all complexity is ultimately about symbolic manipulat…
I think you're mixing up sentience/consciousness/intelligence. Many animals are sentient for example but as far as we know, they don't really have language. But I think I get what you're getting at, which I believe is "human level intelligence requires langauge". I think that's a reasonable take. But you said "sentience", and you said "is", which makes your position difficult to agree with.
Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin
#168With bot-human cooperative text generation this could be very powerful (if the purpose is to generate a large volume of text).
I could tell that something was off when reading through this. The logic didn’t flow and there were contradictions that defeated the previous point. Even more, it wasn’t clear if there was a theme - more like a rambling story.
However, I wouldn’t be surprised if someone told me this was written in a High School English class - as a rough draft.
That is where it occurred to me, that if this was given to a human editor to clean up, it could be interesting.
For example, if writing a creative work of fiction - this could generate a starting point for the human author to refine.
This could be done in an interative process:
- Generate a chunk of text (or multiple chunks). - Edit that into a final text. - Use that as input for the next chunk. - etc.
Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin
#169> I further predict that this will spark a creative gold rush among talented amateurs to train similar models and adapt them to a variety of purposes, including: mock news, “researched journalism”, advertising, politics, and propaganda. The first mention of 'Elon Musk' (who left the board) and this sentence alone gave me the tip-off that GPT-3 had generated that (and the whole blog) and it's following prediction make…
> no-one can train GPT-3 in any acceptable time except Gwern, who has spent probably as much time with GPT-3 and GPT-2 as any 'amateur' out there, is publicly out there saying that for most use cases, GPT-3 + creative use of prompts gets you better results than GPT-2 with finetuning . That's an amazing capability that Gwern elaborates on more here: > A new programming paradigm? The GPT-3 neural network is so large a…
I don't really understand your post (and don't know much about GPT3) are you suggesting that the model is stateful in that it can continue learning from successive prompts?
Maybe you can elaborate on this:
>instead, you interact with it, expressing any task in terms of natural language descriptions, requests, and examples, tweaking the prompt until it “understands” & it meta-learns the new task based on the high-level abstractions it learned from the pretraining.
Or are you suggesting that it has such a deep network of abstractions that once a user starts to map that out, the mileage they can extract back out of the model via prompts is very exciting.
Re: OpenAI's GPT-3 may be the biggest thing since Bitcoin
#170Earlier quoted context omitted.
GPT-3 is a neat party trick. But the things that'll be done with web archives* in the next 20y will make it look like the PDP-8. ~love, a web archivist * GPT-3 is trained on one
The transformer model as presented in GPT-3 may be a few tweaks away from a human-acceptable reasoning, at which point we may realize that human brain is just a neat party trick as well. This may come difficult for some people to internalize, especially those who understand the technology in depth. Because it means that the medium of our reality is the consciousness.
Was this comment generated by GPT-3?