61–70 of 70 posts
Let me guess. GPT-3 wrote this article??
I have some issues with how this article describes GPT and its impact. The scenarios that are described as using GPT-3 remains the stuff of science fiction as far as I know. They would require major technological breakthroughs in areas of autonomous agents, combining language models with 'concept-based' models, and more mundane things like labelling (can you imagine how difficult it would take to accurately label all…
A much more realistic and accurate take on the tech than the article. One combination that can work is to have GPT3 with an unskilled human judge in the loop. The human can discriminate with common sense while the machine generates good jargon with correct grammar.
Jargon (sometimes) isn't just obfuscation; it carries important shades of meaning that would be tedious to spell out every time. 'Homicide' and 'murder', for example, are sometimes used interchangeably, but are not actually legally interchangeable. You would not want your involuntary manslaughter plea upgraded to murder, a more serious--and therefore more severely punished offense.
You could, of course, try to train "unskilled labor" to detect things like that. By the time you are done, the labor won't be unskilled and you will have reinvented the paralegal.
How long until someone builds GPT-4 with several trillion parameters? How much would that cost?
Google has actually already trained a trillion parameter model IIUC [1], though that was a Mixture of Experts so was way cheaper to train.
Earlier quoted context omitted.
I'm not convinced that "capable of reasoning, but not consistently" is a meaningful claim. The examples seem to primarily consist of people spending hours trying things, until eventually GPT-3 outputs a chunk of reasoning they could personally do in seconds. Does that mean that GPT-3 is doing the reasoning, or does it mean that GPT-3 is an English-based lookup table and they managed to find a clever sequence of searc…
Have you seen the database prompt? https://www.gwern.net/GPT-3#the-database-prompt
It appears to be grammatically incorrect. It made me think "ok, probably written by a human." But then also realized that typos and grammar issues are probably prevalent in the data set. How would they manifest? Will they reinforce emerging changes in language? (eg, think of how the meanings of words have slowly mutated). And how much of the scraped content is itself written by a AI?
To me, what GPT-n really tells is how redundant human text in general is. It doesn't provide a lot of new information (if it does, they are accidental), but more or less wandering around the given topic. It is the redundancy that gives the system a space to churn.
GPT is the equivalent, but for language: it can be used to phrase thoughts. Real language however, has thoughts, GPT doesnt have any thoughts. People are impressed by how many responses it has learned, but forget that it is contains a lot of gigabytes of "compressed" text associations. It needs "something else" to become actually useful.
I have seen comments on Reddit that smelled as though something like this was used to create them. I don't see how, in future, we are going to be able to meet up and discuss constructively with strangers online, if this thing can be used to emulate people.
Would it matter if on occasion some value is gleaned from generated content?
I have some issues with how this article describes GPT and its impact. The scenarios that are described as using GPT-3 remains the stuff of science fiction as far as I know. They would require major technological breakthroughs in areas of autonomous agents, combining language models with 'concept-based' models, and more mundane things like labelling (can you imagine how difficult it would take to accurately label all…
A much more realistic and accurate take on the tech than the article. One combination that can work is to have GPT3 with an unskilled human judge in the loop. The human can discriminate with common sense while the machine generates good jargon with correct grammar.
Earlier quoted context omitted.
> Does that mean GPT-3 was trained on an arbitrary, huge database of text? Yes, 500 billion words, for a model with model 175 billion parameters.
So that continues my question: how much of that text would be on the private domain? If there is, is it all legally purchased?