Live data from Hacker News

Yann LeCun on GPT-3

facebook.com

71–80 of 253 posts

Re: Yann LeCun on GPT-3

#71
post #36

Earlier quoted context omitted.

High altitude planes going to the moon is a beautiful analogy. I think this is what I’ll use to explain to less technical friends why I think we’re still many years from self driving cars.

The question isn't whether high-altitude planes can go to the moon, it's whether human intelligence is closer to the clouds or to the moon. For all the talk about how language models "just" learn correlations, there's a remarkable dearth of evidence that humans do something qualitatively different.

> there's a remarkable dearth of evidence that humans do something qualitatively different.

Perhaps now, but if history is any indication, when we (as humans) think we have a good grip on how something really works (like human intellect in this example), we've been wrong.

We model the world around us from observation and testing, find our errors, remodel, and improve over time.

Then at some point we find some piece of information that shows us our model was a decent approximation, but fundamentally wrong, and that we need to start from scratch.

If we find that we want to go beyond the moon (and we eventually will), or that the moon is further than we think, we'll again need a different approach.

I always feel like there's a certain beauty and cosmic humor to it.

Re: Yann LeCun on GPT-3

#72
GPT3 is definitely overrated at this time. Considering how it was built it should not be considered more intelligent than central pattern generators[https://en.wikipedia.org/wiki/Central_pattern_generator]. It's just a pattern generator that generates language instead of a walking pattern. Ascribing to this intelligence has led to some comical claims and studies. Let's start building somethign smart on top of this generator.

Re: Yann LeCun on GPT-3

#73
IMO, the real innovation in GPT-3 is that the API plus "playground" setup is far easier to use than sharing a big chunk of Python code and data files in a Jupyter notebook. If the researchers behind other cutting-edge language models made their work just as easy to use, I believe we would see similar excitement there as well.

Re: Yann LeCun on GPT-3

#74
post #6

For anyone else who doesn’t want to deal with Facebook, here’s the post: Some people have completely unrealistic expectations about what large-scale language models such as GPT-3 can do. This simple explanatory study by my friends at Nabla debunks some of those expectations for people who think massive language models can be used in healthcare. GPT-3 is a language model, which means that you feed it a text and ask it…

As both a GPT-3 sceptic and LeCun-sceptic (nothing personal, I'm just sceptical of a lot of the modern hype and seeming cliquey-ness of the AGI "progress" centres), this summary is very useful, but sounds a bit sour-grapesy from LeCun...

Re: Yann LeCun on GPT-3

#75

Earlier quoted context omitted.

To remind people: Yann LeCun is an engineering superstar who was working on neural networks at Bell labs long before they were cool. Those handwritten digits that are a scourge today (e.g. 'would any of these methods work on a different set of symbols?' is unasked) came from a competition to develop a commercial zip code reader for the U.S. Postal Service post back in the day of the Apple Newton. He won it! I was a b…

To remind people: Yann LeCun worked on artificial neural networks (ANN) during the period where they were actively shunned by most of the scientific community. You could barely publish a paper on ANN. Just to demonstrate, one the most common books during period, "Artificial Intelligence: A Modern Approach, 2nd ed" by Norvig, 1080 pages, has less than one (1!) page dedicated to ANNs. I personally think Norvig is an id…

> Artificial Intelligence: A Modern Approach, 2nd ed

Published in 2002. At that point, ANN research had reached a pretty hard plateau with very few tangible results. Faulting Russel and Norvig for not going into depth about ANNs is kind of like faulting Richard Feynman for not going into depth about quantum computers in the Feynman Lectures.

Also, a lot of the subsequent work and breakthroughs on ANNs has been done at Google under Norvig's leadership as Director of Research.

Re: Yann LeCun on GPT-3

#76

Earlier quoted context omitted.

The humans spent their time building a hideously difficult classification model. Out of the box GPT-3 worked better than the result of a year of their work.

So GPT-3 didn't replace your 2 ML engineers, OpenAI did. GPT-3 didn't build itself.

The iPhone didn't replace your flip phone, apple did. The iPhone didn't build itself.

Re: Yann LeCun on GPT-3

#77

Earlier quoted context omitted.

I have been using that analogy to explain why Tesla is many years away from a self driving car. Several others are building something that is fundamentally different.

Aren't they rolling out a beta of FSD literally as we speak?

I think full self driving will remain two years away for a decade at least.

Re: Yann LeCun on GPT-3

#78

Earlier quoted context omitted.

Can you go into more details where it's useful? As your comment here goes directly against what's argued in the linked Facebook post. Also, if you've found a use case where GPT-3 replaces real humans, what did those humans actually spend their time on? Seems like either you're over-hyping GPT-3, or under-hyping humanity

The humans spent their time building a hideously difficult classification model. Out of the box GPT-3 worked better than the result of a year of their work.

Were there any concerns about GPT-3's latency? It looks like it takes a long time for online use cases.

Re: Yann LeCun on GPT-3

#79
post #2

It's nice to hear from someone who knows what they're talking about that GPT-3 is just a fancy and expensive autocomplete. The hype in some circles about it went as far as comparing it to AGI at some point which is just ridiculous.

You are just a fancy and efficient autocomplete too. When you speak or write, some words have a higher probability than others. You pick alternatives, but they are limited. Of course there are more layers in the human mind, but GPT-3 is a really impressive milestone towards AGI. It's so easy to downplay every advanced tech, it's actually fun. Planes? Just a flying metal tube. Self landing rockets? Just applied physic…

I'm not reducing GPT-3 to the extent that you're suggesting. I'm pointing out (and so does LeCun in his post) that it's a language model designed to continue a sequence of words. It has no understanding of the world and is no particularly suited for knowledge extraction or conversation.

> GPT-3 is a really impressive milestone towards AGI

We really don't know this. It's a big step for the field of language models, that's for sure. But we're so far from AGI that nobody knows which direction it's in and whether it exists at all.

Re: Yann LeCun on GPT-3

#80

Reading this is really interesting: > GPT-3 doesn't have any knowledge of how the world actually works. I think this is a philosophical question. There is a view that, basically, there is no such thing as knowledge, just language (or, at least, there is no distinction between knowledge and language). In this view, all there really is is language, which is mostly composed of metaphors and, ultimately, metaphors only r…

I don't know if i'd go as far as to agree that "there is no knowledge, only language" .. but I 100% agree one of the key insights from GPT-3 -- why training on language is so effective in the first place -- is that language is tightly coupled to reality
Post reply on HN