Yann LeCun on GPT-3

It's nice to hear from someone who knows what they're talking about that GPT-3 is just a fancy and expensive autocomplete. The hype in some circles about it went as far as comparing it to AGI at some point which is just ridiculous.

That is the best summary I have read in a while. Exactly that, only with 3000 words (say) of “prediction”.

For anyone else who doesn’t want to deal with Facebook, here’s the post:

Some people have completely unrealistic expectations about what large-scale language models such as GPT-3 can do.

This simple explanatory study by my friends at Nabla debunks some of those expectations for people who think massive language models can be used in healthcare.

GPT-3 is a language model, which means that you feed it a text and ask it to predict the continuation of the text, one word at a time. GPT-3 doesn't have any knowledge of how the world actually works. It only appears to have some level of background knowledge, to the extent that this knowledge is present in the statistics of text. But this knowledge is very shallow and disconnected from the underlying reality.

As a question-answering system, GPT-3 is not very good. Other approaches that are explicitly built to represent massive amount of knowledge in "neural" associative memories are better at it.

As a dialog system, it's not very good either. Again, other approaches that are explicitly trained to perform to interact with people are better at it.

It's entertaining, and perhaps mildly useful as a creative help. But trying to build intelligent machines by scaling up language models is like a high-altitude airplanes to go to the moon. You might beat altitude records, but going to the moon will require a completely different approach.

It's quite possible that some of the current approaches could be the basis of a good QA system for medical applicatioms. The system could be trained on the entire medical literature and answer questions from physicians. But compiling massive amounts of operational knowledge from text is still very much a research topic.

You can find the study Yann was commenting here https://www.nabla.com/blog/gpt-3/

I would be interested in hearing more about this, within the bounds of what you can share publicly. Most of the touted GPT-3 use cases I've seen to date have dried up or are still in limbo, so hearing about a real production use would be exciting!