In AI parlance, is it correct to say that humans utilise a prompt that is a lifetime's worth of data?
GPT-3 has no idea what it’s talking about
241–250 of 323 posts
Re: GPT-3 has no idea what it’s talking about
#242Earlier quoted context omitted.
For OpenAI to become a healthy and profitable business, GPT-3 will require them to generate ~50-300 million dollars from the model. This could realistically only occur if they cost-effectively fine-tune away the more egregious problems in beta - or convince enough investors that their next model with a 100 million dollar price tag will be able to handle something approximating AGI for realistic applications. This is…
> For OpenAI to become a healthy and profitable business, GPT-3 will require them to generate ~50-300 million dollars from the model. On top of that, does anyone have an idea for what practical applications the model could be used? So far I've only seen the model being used to confuse people; how would one turn that into an ethical business? It seems to me that the "BS route" is indeed the logical course.
Based on the fact GPT-3 seems capable of producing flowery language that is confusing and ultimately nonsensical, it seems to me that GPT-3 has a future writing speeches for politicians ;)
Re: GPT-3 has no idea what it’s talking about
#243Earlier quoted context omitted.
I believe significant share of GPT-3 cost is machine-hours that were spent training this model - months of hundreds top-tier NVidia machines. Edit: estimates range from $2-$5MM to $15MM https://www.reddit.com/r/MachineLearning/comments/hwfjej/d_t...
I thought MS was giving them Azure GPU instances for free?
“Free”, which is basically Microsoft, as an investor, has paid those $4-$15MM minus ~30% of margin they make over pure operational costs.
Electricity was burned and NVidia GPUs that could have been doing paid job were doing GPT-3 training instead.
Re: GPT-3 has no idea what it’s talking about
#244Earlier quoted context omitted.
I think you're kind of proving the OPs point. The argument is that GPT3 has no understanding of the world, just superficial understanding of words and their relationships. If it did have a real understanding, prompt construction wouldn't matter as much, but it clearly does because all GPT3 cares about the structure of sentences, not their meanings.
GPT-3 is a statistical model of text sequences, it has just textual understanding of the world. But the funny thing is that it can do lots of tasks without explicit training, and that is something amazing, it shows a path forward. In order to have real understanding it needs to be an embodied agent that interacts with the world like us, and has goals and needs like us.
The tasks that GPT-3 can do are all language modelling tasks: given a sequence of tokens, what is the most likely sequence of tokens following from it? It's not surprising that this covers tasks like translation and question answering, which are language generation tasks. Other language models have been used in the same manner before, e.g. BERT. I don't remember metrics off the top of my head but I'm not sure GPT-3 is really top notch in many NLP tasks, if any.
In any case a language model is a language model. It has no other ability than calculating the probabilities of sequences of tokens. Leaving aside the question of how something like that can have "real understanding" just by being embodied in the world, how do you even "embody" a language model? I'm genuinely curious to hear how far you have thought about that and how clearly.
I mean in practical terms- you have a trained language model. You have a robotic body (not your own). How do you put them together to produce an embodied agent? What are the intermediarey steps that lead to a robot that can use its language model to... (what does an embodied language model do)?
Re: GPT-3 has no idea what it’s talking about
#245Earlier quoted context omitted.
For OpenAI to become a healthy and profitable business, GPT-3 will require them to generate ~50-300 million dollars from the model. This could realistically only occur if they cost-effectively fine-tune away the more egregious problems in beta - or convince enough investors that their next model with a 100 million dollar price tag will be able to handle something approximating AGI for realistic applications. This is…
> For OpenAI to become a healthy and profitable business, GPT-3 will require them to generate ~50-300 million dollars from the model. On top of that, does anyone have an idea for what practical applications the model could be used? So far I've only seen the model being used to confuse people; how would one turn that into an ethical business? It seems to me that the "BS route" is indeed the logical course.
Entertainment. e.g. $3 per month for My Virtual Friend.
Before you scoff, consider that Pet Rocks were once a (profitable) thing.
Eliza, despite its limitations, sparked considerable engagement with those who were willing to chat with it at length without derailing it.
https://qz.com/1439200/loneliness-costs-the-us-almost-7-bill...
Re: GPT-3 has no idea what it’s talking about
#246Some of the criticism in this comment section is completely fair — the authors are providing exactly the type of prompts that GPT-3 breaks down on and some of these examples might be cherry-picked continuations. And the authors do have personal interests at stake. (NB, the exact same criticism is true about a lot of articles lauding GPT-3, which is why public discussion of GPT-3 in general is such a dumpster fire.) S…
I wonder if you can get GPT-3 bots to spam Reddit, Twitter, and Facebook into oblivion. I also wonder what percentage of users would notice.
Give them a political bent - that's probably what the state actors are trying to productionize right now. Target posts with a sentiment that disagrees with yours, then make the bots follow those users and inundate them with replies wherever they go. Hell, even brands might step in and start doing it.
What's the value of a social network when 25% or more of the comments are from GPT-3 bots?
GPT-3 doesn't need to know anything at all for it to have a very noticeable impact on the web and social media.
Re: GPT-3 has no idea what it’s talking about
#247Earlier quoted context omitted.
Overfitting to the edge case is missing the point. I remind readers of this continuation: > At the party, I poured myself a glass of lemonade, but it turned out to be too sour, so I added a little sugar. I didn’t see a spoon handy, so I stirred it with a cigarette. But that turned out to be a bad idea because it kept falling on the floor. That’s when he decided to start the Cremation Association of North America, whi…
To be fair to the AI, stirring lemonade with a cigarette is so batshit insane that there really can't be a sensible continuation.
tbf to the AI, I quite like the sheer surrealism of its continuation. Though I'd have drawn on my corpus of real world experience with dodgy behaviour around dodgy drinks and gone for 'because I needed to add tequila first'...
But the point isn't the AI can't come up with sensible continuations where sensible continuations aren't obvious, it's the AI will just as readily splice a few strings together in response to silly prompts, they'll just look a bit less impressive than where it's got lots of source material congruent with your expectations to draw upon
Re: GPT-3 has no idea what it’s talking about
#248Earlier quoted context omitted.
Could somebody with GPT-3 access please ask it what words come after "person, woman, man, camera"?
Q: What comes after "person, woman, man, camera" A: person, woman, man, camera, lens, light, film, lab, darkroom. A: person, woman, man, camera, dog, cat, horse A: person, woman, man, camera, camera, camera, camera
I'm not sure what you would expect as a response.
As far as I know, the reference was to a test for dementia in which some words were given at the beginning of the test and asked to be repeated at the end.
Perhaps you could provide context. Maybe there were five words, so you could say "Donald Trump was asked to recall five words to test his memory. Four of them were "person, woman, man, camera". What word did he forget?"
Re: GPT-3 has no idea what it’s talking about
#249Some of the criticism in this comment section is completely fair — the authors are providing exactly the type of prompts that GPT-3 breaks down on and some of these examples might be cherry-picked continuations. And the authors do have personal interests at stake. (NB, the exact same criticism is true about a lot of articles lauding GPT-3, which is why public discussion of GPT-3 in general is such a dumpster fire.) S…
>> And the authors do have personal interests at stake.
Can you say what those are?
Re: GPT-3 has no idea what it’s talking about
#250Some of the criticism in this comment section is completely fair — the authors are providing exactly the type of prompts that GPT-3 breaks down on and some of these examples might be cherry-picked continuations. And the authors do have personal interests at stake. (NB, the exact same criticism is true about a lot of articles lauding GPT-3, which is why public discussion of GPT-3 in general is such a dumpster fire.) S…
I wonder if you can get GPT-3 bots to spam Reddit, Twitter, and Facebook into oblivion. I also wonder what percentage of users would notice. Give them a political bent - that's probably what the state actors are trying to productionize right now. Target posts with a sentiment that disagrees with yours, then make the bots follow those users and inundate them with replies wherever they go. Hell, even brands might step…
I've been wondering about the exact same thing. Basically, a sufficiently calibrated and targeted GPT3 bot swarm could be employed to render at least some parts of these communities into useless echo chambers.
I think that while GPT3 posts are usually identifiable after reading some sentences, I often also find that it's harder to realize when consuming precisely this kind of social media. I often just consume it absent mindedly as opposed to reading a real article.