Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Yann LeCun on GPT-3

facebook.com

Post reply on HN
Print view View original

61–70 of 253 posts

Re: Yann LeCun on GPT-3

#61

Post by ibarea23 »

thomas_rm wrote: post #49
PaulHoule wrote: post #33

Earlier quoted context omitted.

To remind people: Yann LeCun is an engineering superstar who was working on neural networks at Bell labs long before they were cool. Those handwritten digits that are a scourge today (e.g. 'would any of these methods work on a different set of symbols?' is unasked) came from a competition to develop a commercial zip code reader for the U.S. Postal Service post back in the day of the Apple Newton. He won it! I was a b…

To remind people: Yann LeCun worked on artificial neural networks (ANN) during the period where they were actively shunned by most of the scientific community. You could barely publish a paper on ANN. Just to demonstrate, one the most common books during period, "Artificial Intelligence: A Modern Approach, 2nd ed" by Norvig, 1080 pages, has less than one (1!) page dedicated to ANNs. I personally think Norvig is an id…

Norvig is not an idiot, and one of the humblest and nicest people I've ever met. A lot of people were wrong about ANNs.

Re: Yann LeCun on GPT-3

#62

Post by hooande »

solveit wrote: post #36
formercoder wrote: post #16

Earlier quoted context omitted.

High altitude planes going to the moon is a beautiful analogy. I think this is what I’ll use to explain to less technical friends why I think we’re still many years from self driving cars.

The question isn't whether high-altitude planes can go to the moon, it's whether human intelligence is closer to the clouds or to the moon. For all the talk about how language models "just" learn correlations, there's a remarkable dearth of evidence that humans do something qualitatively different.

Are you just dumbing down humans to match the model? LeCunn's post is very sparse on detail, but the point is that humans can easily reason about a vast number of things that any form of sequential language model cannot. That alone is evidence that humans are doing something qualitatively different.

It isn't conclusive evidence however, and larger models may produce significantly more human like results. But from what we know about how gpt-3 works, all the evidence is on the side of it not resembling human intelligence.

Re: Yann LeCun on GPT-3

#63

Post by luc4sdreyer »

leftyted wrote: post #18

Reading this is really interesting: > GPT-3 doesn't have any knowledge of how the world actually works. I think this is a philosophical question. There is a view that, basically, there is no such thing as knowledge, just language (or, at least, there is no distinction between knowledge and language). In this view, all there really is is language, which is mostly composed of metaphors and, ultimately, metaphors only r…

At the risk of reigniting the perpetual war about how to characterize machine intelligence, and by extension how to characterize the risk they pose, Yann has been (and still is AFAIK) more in the "existential AI risk is a long-term problem" group. In a 2016 interview LeCun said [1]:

> We’re very far from having machines that can learn the most basic things about the world in the way humans and animals can do. Like, yes, in particular areas machines have superhuman performance, but in terms of general intelligence we’re not even close to a rat. This makes a lot of questions people are asking themselves premature. . That’s not to say we shouldn’t think about them, but there’s no danger in the immediate or even medium term. There are real dangers in the department of AI, real risks, but they’re not Terminator scenarios.

That's pretty measured overall, but he doesn't know that there's no existential AI risk in the medium term. No one does, and that's the problem. Experts simply suspect that it's unlikely. Stuart Russell and him have debated similar topics [2].

To tie back to your point: I keep seeing LeCun brush over tricky questions like yours and the ones at [2] with an arrogant confidence. I wish that he would be more careful, and I hope that I have a skewed view of him.

[1] https://www.theverge.com/2017/10/26/16552056/a-intelligence-...

[2] https://www.lesswrong.com/posts/WxW6Gc6f2z3mzmqKs/debate-on-...

Re: Yann LeCun on GPT-3

#64

Post by cbozeman »

PaulHoule wrote: post #33
czzr wrote: post #6

For anyone else who doesn’t want to deal with Facebook, here’s the post: Some people have completely unrealistic expectations about what large-scale language models such as GPT-3 can do. This simple explanatory study by my friends at Nabla debunks some of those expectations for people who think massive language models can be used in healthcare. GPT-3 is a language model, which means that you feed it a text and ask it…

To remind people: Yann LeCun is an engineering superstar who was working on neural networks at Bell labs long before they were cool. Those handwritten digits that are a scourge today (e.g. 'would any of these methods work on a different set of symbols?' is unasked) came from a competition to develop a commercial zip code reader for the U.S. Postal Service post back in the day of the Apple Newton. He won it! I was a b…

If you think Peter Thiel spouts nonsense, you must really be a 300 IQ megabrain...

Re: Yann LeCun on GPT-3

#65

Post by leftyted »

alexanderskates wrote: post #52
leftyted wrote: post #18

Reading this is really interesting: > GPT-3 doesn't have any knowledge of how the world actually works. I think this is a philosophical question. There is a view that, basically, there is no such thing as knowledge, just language (or, at least, there is no distinction between knowledge and language). In this view, all there really is is language, which is mostly composed of metaphors and, ultimately, metaphors only r…

I think an important distinction to make is your use of the word "language", and how we think of language as it concerns human minds, and as it concerns GPT-3. In our heads, language is a combination of words and concepts, and knowledge can be encoded by making connections between concepts , not simply words. If there is no concept or idea backing up the words, it can hardly be called knowledge. Consider the case of…

> In our heads, language is a combination of words and concepts, and knowledge can be encoded by making connections between concepts, not simply words. If there is no concept or idea backing up the words, it can hardly be called knowledge.

Great point.

> A language model such as GPT-3 operates only on words, not concepts. It can make connections between words on the basis of statistical correlations, but has no capacity for encoding concepts, and therefore cannot "know" anything.

Are you sure? Aren't "concepts" encoded in how language is used, at least to some degree?

LeCun does say that models that explicitly attempt represent knowledge perform better than GPT-3 in terms of answering questions. I'm no expert but I believe him.

Re: Yann LeCun on GPT-3

#66

Post by this_was_posted »

To me GPT-3 feels more like a rocket-booster than a high-altitude plane. On its own it's not going to reach the moon, but combined with the right guidance and additional thrust it just might.

I think being able to model future outcome of something in a similar way humans would (like GPT-3 does) is the first input step for an overarching AI to reach some kind of sentience.

With my admittedly limited understanding I believe that what differentiates our thinking most from other animals is that we are able to evaluate, order and steer our thoughts much better. If we can develop something that can steer these GPT-3 "thoughts" I imagine we could get quite close to sentience

Re: Yann LeCun on GPT-3

#67

Post by amelius »

This doesn't sound like a very rigorous refutation. Is this the way debunking works in deep learning circles?

Anyway, I can refute the refutal using the same standard: lots of things about the real world can be learned from just reading text, and there is no reason given why a DL model couldn't too.

Re: Yann LeCun on GPT-3

#68

Post by integer42 »

cbozeman wrote: post #64
PaulHoule wrote: post #33

Earlier quoted context omitted.

To remind people: Yann LeCun is an engineering superstar who was working on neural networks at Bell labs long before they were cool. Those handwritten digits that are a scourge today (e.g. 'would any of these methods work on a different set of symbols?' is unasked) came from a competition to develop a commercial zip code reader for the U.S. Postal Service post back in the day of the Apple Newton. He won it! I was a b…

If you think Peter Thiel spouts nonsense, you must really be a 300 IQ megabrain...

Or you're just a regular person who can detect silicon valley flavored self help platitudes

Re: Yann LeCun on GPT-3

#69

Post by tephra »

thomas_rm wrote: post #49
PaulHoule wrote: post #33

Earlier quoted context omitted.

To remind people: Yann LeCun is an engineering superstar who was working on neural networks at Bell labs long before they were cool. Those handwritten digits that are a scourge today (e.g. 'would any of these methods work on a different set of symbols?' is unasked) came from a competition to develop a commercial zip code reader for the U.S. Postal Service post back in the day of the Apple Newton. He won it! I was a b…

To remind people: Yann LeCun worked on artificial neural networks (ANN) during the period where they were actively shunned by most of the scientific community. You could barely publish a paper on ANN. Just to demonstrate, one the most common books during period, "Artificial Intelligence: A Modern Approach, 2nd ed" by Norvig, 1080 pages, has less than one (1!) page dedicated to ANNs. I personally think Norvig is an id…

For some context. The 2nd edition was published in 2002 (so maybe written in 2001-2002?). The fourth edition published in 2020 seems to have a bunch more things on NNs.

Re: Yann LeCun on GPT-3

#70

Post by lacker »

thomas_rm wrote: post #49
PaulHoule wrote: post #33

Earlier quoted context omitted.

To remind people: Yann LeCun is an engineering superstar who was working on neural networks at Bell labs long before they were cool. Those handwritten digits that are a scourge today (e.g. 'would any of these methods work on a different set of symbols?' is unasked) came from a competition to develop a commercial zip code reader for the U.S. Postal Service post back in the day of the Apple Newton. He won it! I was a b…

To remind people: Yann LeCun worked on artificial neural networks (ANN) during the period where they were actively shunned by most of the scientific community. You could barely publish a paper on ANN. Just to demonstrate, one the most common books during period, "Artificial Intelligence: A Modern Approach, 2nd ed" by Norvig, 1080 pages, has less than one (1!) page dedicated to ANNs. I personally think Norvig is an id…

It is not fair to call Norvig an idiot. When I was studying AI in grad school, around 2002-2003, just about everyone thought that artificial neural networks were a dead end, compared to approaches like support vector machines. Sometimes the scientific consensus is wrong, and it takes a few heroic figures plugging away to prove it. That doesn't mean that everyone in the mainstream is an "idiot".
Post reply on HN