I wonder what he thinks about using it in automated theorem proving: https://twitter.com/spolu/status/1303578985276887042
Yann LeCun on GPT-3
31–40 of 253 posts
Re: Yann LeCun on GPT-3
#32For anyone else who doesn’t want to deal with Facebook, here’s the post: Some people have completely unrealistic expectations about what large-scale language models such as GPT-3 can do. This simple explanatory study by my friends at Nabla debunks some of those expectations for people who think massive language models can be used in healthcare. GPT-3 is a language model, which means that you feed it a text and ask it…
I agree that GPT-3 should not be used for medical applications, but I disagree that it's "not very good" as a dialog system. I've found it to be insanely good though it may require effective prompt engineering to work well.
Re: Yann LeCun on GPT-3
#33For anyone else who doesn’t want to deal with Facebook, here’s the post: Some people have completely unrealistic expectations about what large-scale language models such as GPT-3 can do. This simple explanatory study by my friends at Nabla debunks some of those expectations for people who think massive language models can be used in healthcare. GPT-3 is a language model, which means that you feed it a text and ask it…
To remind people: Yann LeCun is an engineering superstar who was working on neural networks at Bell labs long before they were cool.
Those handwritten digits that are a scourge today (e.g. 'would any of these methods work on a different set of symbols?' is unasked) came from a competition to develop a commercial zip code reader for the U.S. Postal Service post back in the day of the Apple Newton. He won it!
I was a bagman for text classification data in the early 2000's and his reviews of the results you got using methods of the time (Naive Bayes, Rocchio, Perceptron, SVM) showed a depth of thought and attention to detail which helped me pick and choose tools to make classifiers with fairly predictable performance and development paths.
GPT-3 on the other hand does a good job of spouting nonsense like Peter Thiel and that has something to do with it's emotional appeal. People make fun of it and laugh at the mistakes it makes like those videos where somebody kicks down one of those Boston Robotics dogs: it's just good enough to be an object for those sort of feelings.
Re: Yann LeCun on GPT-3
#34One thing I've been wondering, could you train a GPT-3 model to generate "better" text prompts for another GPT-3 model
By better I mean grading based on whether there is any nonsense in the output or any internal contradictions, or similar criteria
Re: Yann LeCun on GPT-3
#35Reading this is really interesting: > GPT-3 doesn't have any knowledge of how the world actually works. I think this is a philosophical question. There is a view that, basically, there is no such thing as knowledge, just language (or, at least, there is no distinction between knowledge and language). In this view, all there really is is language, which is mostly composed of metaphors and, ultimately, metaphors only r…
Animals that do not have a language they can describe the world in still have knowledge about the world.
Personally I do not find the whole "language = knowledge" argument convincing. But if you're interested in reading writers who make that argument (and perhaps I'm vulgarizing the argument a bit), Nietzsche makes it in On Truth and Falsity in their Extra-Moral Sense and George Lakoff makes it in Metaphors We Live By.
Re: Yann LeCun on GPT-3
#36For anyone else who doesn’t want to deal with Facebook, here’s the post: Some people have completely unrealistic expectations about what large-scale language models such as GPT-3 can do. This simple explanatory study by my friends at Nabla debunks some of those expectations for people who think massive language models can be used in healthcare. GPT-3 is a language model, which means that you feed it a text and ask it…
High altitude planes going to the moon is a beautiful analogy. I think this is what I’ll use to explain to less technical friends why I think we’re still many years from self driving cars.
The question isn't whether high-altitude planes can go to the moon, it's whether human intelligence is closer to the clouds or to the moon. For all the talk about how language models "just" learn correlations, there's a remarkable dearth of evidence that humans do something qualitatively different.
Re: Yann LeCun on GPT-3
#37> Some people have completely unrealistic expectations about what large-scale language models such as GPT-3 can do.
Just want to point out that he's saying the people on the upper end of the expectation distribution are wrong, not the people in the middle of it. So if you're takeaway from this is that GPT3 is nothing special, that's probably the wrong message.
Re: Yann LeCun on GPT-3
#38It's nice to hear from someone who knows what they're talking about that GPT-3 is just a fancy and expensive autocomplete. The hype in some circles about it went as far as comparing it to AGI at some point which is just ridiculous.
You're correct. It's only autocomplete on steroids. But I think it's remarkable that something with the very simple goal of autocomplete can, for a few sentences, sound almost alive
Re: Yann LeCun on GPT-3
#39It's nice to hear from someone who knows what they're talking about that GPT-3 is just a fancy and expensive autocomplete. The hype in some circles about it went as far as comparing it to AGI at some point which is just ridiculous.
What evidence do I have that I'm more than a fancy autocomplete, myself? The use of squishy protestations, in lieu of objective metrics, make LeCun's argument rather unconvincing.
I was about to write a reply claiming that you're different from autocomplete because you take input from more sources than just the words you've said before (e.g. your vision), but actually I can't see how that's much different from a language model. The approach seems the same, and all that's really different is the shape of the input data.
But this uncovers difficult questions about free will. If we're all just autocompleting based on a combination of the world around us, our internal state, and the physical laws, then what even is intelligence anyway? This view reduces thought to nothing more than an interesting dust storm.
Still, I find the original argument compelling, if not logically convincing. There does seem to be something missing from GPT-3 that differs fundamentally from human intelligence or AGI. But maybe that's an illusion.
Edit: I don't think you should have been downvoted, since your question is valid and constructive in my view.
Re: Yann LeCun on GPT-3
#40Earlier quoted context omitted.
Can you go into more details where it's useful? As your comment here goes directly against what's argued in the linked Facebook post. Also, if you've found a use case where GPT-3 replaces real humans, what did those humans actually spend their time on? Seems like either you're over-hyping GPT-3, or under-hyping humanity
The humans spent their time building a hideously difficult classification model. Out of the box GPT-3 worked better than the result of a year of their work.
So GPT-3 didn't replace your 2 ML engineers, OpenAI did.
GPT-3 didn't build itself.