291–300 of 323 posts
Earlier quoted context omitted.
I thought MS was giving them Azure GPU instances for free?
“Free”, which is basically Microsoft, as an investor, has paid those $4-$15MM minus ~30% of margin they make over pure operational costs. Electricity was burned and NVidia GPUs that could have been doing paid job were doing GPT-3 training instead.
Earlier quoted context omitted.
Simple Markov Chains of the sort you might assign as an undergrad programming assignment can write impressive poetry/captions if you tweak the inputs and cherry-pick outputs. There’s a whole Reply All episode of tech journo types being wowed by 90s text generation tech. Nothing wrong with that; it is what it is. But, do markov chains do few-shot learning? What’s actually unclear to me that there is much economic/scie…
What’s the difference between careful prompt design and any other type of careful design?
Earlier quoted context omitted.
I actually just sent out an API request for a particular application that I believe GPT-3 should be capable. I'm a university mathematics instructor. I find that a lot of students struggle with doing proofs and I think GPT-3 can help with that. A proof is essentially a sequence of logical inferences. I believe that, when given a proof written in natural language GPT-3 should be capable of detecting the logical implic…
Really? This seems like the exact sort of thing GPT-3 would be very bad at. The statements aren't the product of an internally consistent logical model, they are statistically plausible word sequences learned in a really clever way from a huge corpus of text. That said, before dismissing it sight unseen there are effective and quick ways to test its powers with mathematics. You could prime an instance with the axioms…
It could generate math problems and have students try them out. Might be a bit of fun.
Why must we keep having this argument? If you do research in the field you know full well that GPT/any other transformer or Bert model is generating text by regurgitating approximate conditional probabilities of words given all the text it has ever seen and the prompt. The neurophysiological concept of “understanding” as most understand it is orthogonal to the way the algorithm actually works. A more useful conversat…
That would be eminently useful, but unfortunately we can't have that discussion because OpenAI aren't exposing the model.
They've really brought this on themselves - I don't think there'd be these believer/nonbeliever camps if they had taken the slower, rationalist/scientific approach to the research.
Instead, they've breathlessly hyped up their new API with media releases and saturated social media, and are picking and choosing who they allow to play with their model. It's not surprising that a lot of people didn't take too kindly to it.
Earlier quoted context omitted.
Overfitting to the edge case is missing the point. I remind readers of this continuation: > At the party, I poured myself a glass of lemonade, but it turned out to be too sour, so I added a little sugar. I didn’t see a spoon handy, so I stirred it with a cigarette. But that turned out to be a bad idea because it kept falling on the floor. That’s when he decided to start the Cremation Association of North America, whi…
Tried a GPT-3 completion: > At the party, I poured myself a glass of lemonade, but it turned out to be too sour, so I added a little sugar. I didn’t see a spoon handy, so I stirred it with a cigarette. But that turned out to be a bad idea because ... > ... the cigarette burned a hole in the tablecloth. I didn’t have another one, so I used my finger to stir it and then licked my finger. > After that, I went into the l…
Earlier quoted context omitted.
- ...And Morris? - Um, it actually already is a word. Tnetennba. - Good heavens, really? Could you, uh, use it in a sentence for us? - "Good morning. That's a nice Tnetennba" [1]. ______________ [1] Moss from IT Crowd on Countdown: https://youtu.be/g9ixvD0_CmM?t=52 Edit: to clarify, if you don't know what a word means, just seeing it used in a sentence won't necessarily tell you much about its meaning, so that a lang…
Maybe not, but the fact that it actually made a sentence with a word in it that it could not have possibly seen in the training data tells us that it understood something about the meaning of the instructions.
In any case, it's a language model. It has no ability to "understand" anything. It can compute the probability of a token to follow from a sequence of tokens, and that's all. There's no "understanding" there, nobody made it to understand anything.
Earlier quoted context omitted.
> For OpenAI to become a healthy and profitable business, GPT-3 will require them to generate ~50-300 million dollars from the model. On top of that, does anyone have an idea for what practical applications the model could be used? So far I've only seen the model being used to confuse people; how would one turn that into an ethical business? It seems to me that the "BS route" is indeed the logical course.
Based on the fact GPT-3 seems capable of producing flowery language that is confusing and ultimately nonsensical, it seems to me that GPT-3 has a future writing speeches for politicians ;)
Aaaw! Eugene Charniak is one of my heroes of AI, after I read his little green book, Statistical Language Learning [1] during my Masters. It remains a great resource for a quick and dirty, but thorough and broad introduction to the field of statistical NLP that goes through all the basics.
In fact, now that I think about it, if more people read that little book (it's only 199 pages) we would have many fewer discussions about how GPT-3 "understands" or "knows" etc.
Anyway, thanks to Gary marcus for pointing out Charniak's thesis which I hadn't read.
____________
[1] https://mitpress.mit.edu/books/statistical-language-learning
Earlier quoted context omitted.
> For OpenAI to become a healthy and profitable business, GPT-3 will require them to generate ~50-300 million dollars from the model. On top of that, does anyone have an idea for what practical applications the model could be used? So far I've only seen the model being used to confuse people; how would one turn that into an ethical business? It seems to me that the "BS route" is indeed the logical course.
Entertainment. e.g. $3 per month for My Virtual Friend. Before you scoff, consider that Pet Rocks were once a (profitable) thing. Eliza, despite its limitations, sparked considerable engagement with those who were willing to chat with it at length without derailing it. https://qz.com/1439200/loneliness-costs-the-us-almost-7-bill...