I think we have to make a distiction here:
- On one hand, having access to these large scale language models that can do few-shot learning is incredibly useful for the industry as in can be easily deployed to solve thosands of simple tasks.
- On the other hand, this approach will not solve harder problems (as Yann points out) and "just" creating bigger models using the same techniques is probably not the path forward in those domains.
Yann LeCun on GPT-3
251–253 of 253 posts
Re: Yann LeCun on GPT-3
#252Reading this is really interesting: > GPT-3 doesn't have any knowledge of how the world actually works. I think this is a philosophical question. There is a view that, basically, there is no such thing as knowledge, just language (or, at least, there is no distinction between knowledge and language). In this view, all there really is is language, which is mostly composed of metaphors and, ultimately, metaphors only r…
> metaphors only refer to other metaphors, i.e. language is circular.
Except we (humans) have real-life experience that gives meaning to those metaphors.
Re: Yann LeCun on GPT-3
#253Really? I have been doing research on language models in medical diagnostics even before GPT-2, and found that when trained and applied in certain ways, language models (even much smaller than GPT-3!) are very good at diagnosis predictions, they can compete with much more complex symptom checkers at that. Proof: Link to my paper (written back in 2019) and a bit less technical article. http://www.dialog-21.ru/media/46…
I used GPT-2 to create a health website. One sentence was enough to get a full page of authoritatively sounding lists of symptoms and treatments. Very diverse, unlike all other sites, because the articles it generated only looked and sounded like a health encyclopedia. Of course it is going to spit back decent diagnosis, when it is in the training data, but what do you trust? An expert system that logically and inter…
It is self-supervised learning. Specifically, a masking auto encoder.