Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Yann LeCun on GPT-3

facebook.com

Post reply on HN
Print view View original

181–190 of 253 posts

Re: Yann LeCun on GPT-3

#181

Post by powerapple »

alexanderskates wrote: post #52
leftyted wrote: post #18

Reading this is really interesting: > GPT-3 doesn't have any knowledge of how the world actually works. I think this is a philosophical question. There is a view that, basically, there is no such thing as knowledge, just language (or, at least, there is no distinction between knowledge and language). In this view, all there really is is language, which is mostly composed of metaphors and, ultimately, metaphors only r…

I think an important distinction to make is your use of the word "language", and how we think of language as it concerns human minds, and as it concerns GPT-3. In our heads, language is a combination of words and concepts, and knowledge can be encoded by making connections between concepts , not simply words. If there is no concept or idea backing up the words, it can hardly be called knowledge. Consider the case of…

Language is how we store our knowledge, and language is a system of words. If a language model contains all the possible sentences you can say, it will complete any of your sentences, don't you think it knows what you know? The input is sequence of characters, so you can say it may or may not operate on words. It can operate on subwords, words or phrases where it see fit. I like to think intelligence as clouds. If you dig deep down, they are just droplets, there are so many of them, they can appear to be so many different shapes. And they look complete different. Maybe intelligence is the same.

Re: Yann LeCun on GPT-3

#182

Post by alper111 »

alper111 wrote: post #138

It is quite interesting that LeCun is very critical when it comes to GPT from OpenAI. The same arguments can also be said for the current deep learning paradigm and convolutional nets, but you don't see any criticism from him when it comes to this stuff. Look at his arguments when he is tweet-debating with Gary Marcus.

I also do not understand the downvote for a fair criticism.

Re: Yann LeCun on GPT-3

#183

Post by jimbokun »

fludlight wrote: post #88
neural_thing wrote: post #5

I'm sure his group has done some rigorous research that I can't even understand. But in my experience, the few-shot learner attribute of GPT-3 makes it insanely useful. We have already found several use cases for it, one of which replaces 2 ML engineers. Yes, it's not perfect, but it's pretty good at many things, and REALLY easy to use.

“AI” replacing the jobs of AI engineers. But we were told it was only going to do that to blue collar work!

We need UBI yesterday.

Re: Yann LeCun on GPT-3

#184

Post by powerapple »

erispoe wrote: post #22
leftyted wrote: post #18

Reading this is really interesting: > GPT-3 doesn't have any knowledge of how the world actually works. I think this is a philosophical question. There is a view that, basically, there is no such thing as knowledge, just language (or, at least, there is no distinction between knowledge and language). In this view, all there really is is language, which is mostly composed of metaphors and, ultimately, metaphors only r…

Animals that do not have a language they can describe the world in still have knowledge about the world.

Are animals as intelligent as human? What language does to us is to be able to pass on knowledge to generations, knowledge can be accumulated. Animals may be able to pass on simple concept to next generation, they won't be able to accumulate knowledge without language and writing.

Re: Yann LeCun on GPT-3

#185

Post by thomas_rm »

lisper wrote: post #154
thomas_rm wrote: post #144

Earlier quoted context omitted.

We seem to be re-enacting the Symbolic vs Connectionist AI debate of the 80s, poorly. All I'm saying is, Norvig should have been more humble and included a chapter or two about ANNs, with all the research accumulated thus far, instead of betting 100% for the symbolic approach. Let the next generation of students learn both approaches and decide for themselves. It's sad that a whole generation of students was taught A…

> All I'm saying is, Norvig should have been more humble and included a chapter or two about ANNs No, that is not all you're saying. You opened with this: "I personally think Norvig is an idiot with regards to Artificial Intelligence," Not only did you lob an ad hominem at one of the most respected members of the community simply for making an editorial decision 18 years ago that you happen not to agree with today, y…

I think Norvig is a smart person, I enjoy his books, papers and jupyter notebooks, but I always thought he was pretty clueless regarding AI, as history indeed demonstrated. That's not an ad hominem.

What is shameful was the extreme shunning of the mainstream scientific community to ANNs, in 90's-00's decades, to the point that it was considered career suicide to publish a ANN paper. I believe Yann LeCun has said similar things in the past[0], reminiscing the time when it took him several years(!) to get an ANN paper accepted for publication.

[0] http://yann.lecun.com/ex/pamphlets/publishing-models.html

Re: Yann LeCun on GPT-3

#186

Post by ivalm »

neural_thing wrote: post #15
digital-cygnet wrote: post #10

Earlier quoted context omitted.

I would be interested in hearing more about this, within the bounds of what you can share publicly. Most of the touted GPT-3 use cases I've seen to date have dried up or are still in limbo, so hearing about a real production use would be exciting!

Extremely complex classification task is all I can say

In general GPT3 is not SotA on (any?) classification task, did you just not have enough data to fine tune a discriminative transformer model? Inference should be cheaper with a smaller transformer/also less lock-in.

Re: Yann LeCun on GPT-3

#187

Post by neural_thing »

ivalm wrote: post #186
neural_thing wrote: post #15

Earlier quoted context omitted.

Extremely complex classification task is all I can say

In general GPT3 is not SotA on (any?) classification task, did you just not have enough data to fine tune a discriminative transformer model? Inference should be cheaper with a smaller transformer/also less lock-in.

I can't go into too much detail here about why we couldn't do that, but one aspect that we found VERY useful is that GPT-3 could draw on real world knowledge not present in the dataset to enhance the results.

Re: Yann LeCun on GPT-3

#188

Post by disgruntledphd2 »

aaaxyz wrote: post #143
whymauri wrote: post #114

Earlier quoted context omitted.

Because most "AI engineering" has lost its meaning and is actually data analysis.

Did it ever have a single meaning? Every company I've been through had a different definition of what "AI engineering" should be

AI engineering: a means of extraction of venture capital from gullible investors :)

Mostly sarcasm, mostly.

Re: Yann LeCun on GPT-3

#189

Post by fastball »

arugulum wrote: post #175
fastball wrote: post #149

Earlier quoted context omitted.

I think the onus is on you to prove that the use cases are shallow and limited. I've seen GPT-3 already being used for diverse and interesting ideas that would not have occurred to me personally. However, even if they are, the point stands: currently, there are teams of people at companies all over the world tuning models for these shallow and limited use-cases. GPT-3 can replace them all, without OpenAI needing to i…

>However, even if they are, the point stands: currently, there are teams of people at companies all over the world tuning models for these shallow and limited use-cases. GPT-3 can replace them all, without OpenAI needing to invest another cent in training for a particular customer's use-case. That is in fact game-changing for the ML/DL world and current applications thereof. The counterpoint is that it would be signi…

Clearly that is not true for the commenter that started this thread.

Re: Yann LeCun on GPT-3

#190

Post by mannykannot »

powerapple wrote: post #184
erispoe wrote: post #22

Earlier quoted context omitted.

Animals that do not have a language they can describe the world in still have knowledge about the world.

Are animals as intelligent as human? What language does to us is to be able to pass on knowledge to generations, knowledge can be accumulated. Animals may be able to pass on simple concept to next generation, they won't be able to accumulate knowledge without language and writing.

As a response to leftyted, erispoe's point neither depends on nor implies that animals are as intelligent as humans, it is simply bringing up a counter-example: what appears to be knowledge in animals lacking language.

Of course, you could always attempt to define knowledge such that it is purely verbal, or alternatively define whatever is going on in the brain of an animal to be language, but is either approach useful? In common usage, we recognise, as knowledge, various things that cannot be communicated by language, such as knowing how to ride a unicycle on a tightrope (I doubt you can learn it just from a book) and the infamous qualia which supposedly prove that the mind is dualistic. And what about the knowledge of how to use language? How does that get bootstrapped?

Post reply on HN