Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Yann LeCun on GPT-3

facebook.com

Post reply on HN
Print view View original

191–200 of 253 posts

Re: Yann LeCun on GPT-3

#191

Post by c06n »

powerapple wrote: post #184
erispoe wrote: post #22

Earlier quoted context omitted.

Animals that do not have a language they can describe the world in still have knowledge about the world.

Are animals as intelligent as human? What language does to us is to be able to pass on knowledge to generations, knowledge can be accumulated. Animals may be able to pass on simple concept to next generation, they won't be able to accumulate knowledge without language and writing.

In important ways yes, and in important ways no. Yes: can deal with dynamic spatial environments, can act towards a goal (intentionality), some almost certainly can plan how to reache these goals. No: Use of symbols and recursive language.

Re: Yann LeCun on GPT-3

#192

Post by disgruntledphd2 »

GrantZvolsky wrote: post #56
czzr wrote: post #6

For anyone else who doesn’t want to deal with Facebook, here’s the post: Some people have completely unrealistic expectations about what large-scale language models such as GPT-3 can do. This simple explanatory study by my friends at Nabla debunks some of those expectations for people who think massive language models can be used in healthcare. GPT-3 is a language model, which means that you feed it a text and ask it…

> GPT-3 doesn't have any knowledge of how the world actually works. It only appears to have some level of background knowledge, to the extent that this knowledge is present in the statistics of text. But this knowledge is very shallow and disconnected from the underlying reality. Without excessive effort, humans don't have any knowledge of how the world actually works. They only appear to have some level of backgroun…

This is just not true.

For example, all humans have the notion of object permanence, developed by about six months of life. Object permanence is the notion that things don't go away just because you can't see them.

ML systems need to be specifically trained to have object permanence, and GPT-3 almost certainly does not possess it.

Like, I get that it's hip to booster ML and GPT-3, and all of the stuff humans can do seem trivial, but it's really not the case and is something that is holding progress in AI back massively.

Re: Yann LeCun on GPT-3

#193

Post by disgruntledphd2 »

CJefferson wrote: post #171
vessenes wrote: post #83

I have a really hard time believing that LeCun thinks this is true. GPT3 is outstanding at conversation. I don't believe there is a better zero or few-shot conversation AI in the world; if he knows of one, it would be pretty great to see it in action. Here is a conversation I had with my GPT-3 chatbot a few months ago. It is cherry picked in the sense that I have had non-sensical conversations as well, but I did not…

It depends on how you define shallow. I find GPT3 is indeed amazing at general knowledge, and making plausible sentences and conversations out of that general knowledge. However, if I start asking it questions which probably haven't have been asked before, or to connect ideas, it often falls apart.

GPT is great at text generation, but astonishingly poor at coherent text generation over a a window of more than a few sentences.

To be fair, this is a massive improvement from SOTA ten years ago, but it's definitely nowhere near AGI (and maybe not even a finger pointing at the moon).

Re: Yann LeCun on GPT-3

#194

Post by sgt3pr »

leftyted wrote: post #18

Reading this is really interesting: > GPT-3 doesn't have any knowledge of how the world actually works. I think this is a philosophical question. There is a view that, basically, there is no such thing as knowledge, just language (or, at least, there is no distinction between knowledge and language). In this view, all there really is is language, which is mostly composed of metaphors and, ultimately, metaphors only r…

I find this concept of knowledge encoded by language very interesting. Is there any author you can point to that follows this idea?

Re: Yann LeCun on GPT-3

#195

Post by jsinai »

leftyted wrote: post #65
alexanderskates wrote: post #52

Earlier quoted context omitted.

I think an important distinction to make is your use of the word "language", and how we think of language as it concerns human minds, and as it concerns GPT-3. In our heads, language is a combination of words and concepts, and knowledge can be encoded by making connections between concepts , not simply words. If there is no concept or idea backing up the words, it can hardly be called knowledge. Consider the case of…

> In our heads, language is a combination of words and concepts, and knowledge can be encoded by making connections between concepts, not simply words. If there is no concept or idea backing up the words, it can hardly be called knowledge. Great point. > A language model such as GPT-3 operates only on words, not concepts. It can make connections between words on the basis of statistical correlations, but has no capac…

>Aren’t “concepts” encoded in how language is used, at least to some degree?

Good point and I think this shows up to the extent different languages might affect how we express particular concepts.

However I think it is more accurate to say that language solidifies and gives form to how we express concepts and the “concepts” themselves are independent of languages. Only our “expression” of these “concepts” depends on language.

For anyone interested in art and art history, this distinction was the central focus of the French surrealist painter Rene Magritte.

Re: Yann LeCun on GPT-3

#196

Post by jart »

leftyted wrote: post #18

Reading this is really interesting: > GPT-3 doesn't have any knowledge of how the world actually works. I think this is a philosophical question. There is a view that, basically, there is no such thing as knowledge, just language (or, at least, there is no distinction between knowledge and language). In this view, all there really is is language, which is mostly composed of metaphors and, ultimately, metaphors only r…

It's not that philosophical. If you visualize the GPT-3 embeddings using an embeddings projector then you can see the knowledge with your own eyes. That bypasses the need for GPT-3 to use language to communicate its knowledge to you. Computers aren't a black box like we've traditionally thought of brains.

Re: Yann LeCun on GPT-3

#197

Post by nurettin »

leftyted wrote: post #18

Reading this is really interesting: > GPT-3 doesn't have any knowledge of how the world actually works. I think this is a philosophical question. There is a view that, basically, there is no such thing as knowledge, just language (or, at least, there is no distinction between knowledge and language). In this view, all there really is is language, which is mostly composed of metaphors and, ultimately, metaphors only r…

Can language as knowledge be falsified? Or is every statement equally true?

Re: Yann LeCun on GPT-3

#198

Post by ardit33 »

wongarsu wrote: post #132
fludlight wrote: post #88

Earlier quoted context omitted.

“AI” replacing the jobs of AI engineers. But we were told it was only going to do that to blue collar work!

Better IDEs have saved countless hours. Saving hours is equivalent to replacing jobs, unless demand is elastic enough to fill that time. Most of the time we are lucky enough that demand at a given price point is much larger than supply, but this won't last forever.

that's not how it works... as it is not a zero sum (i.e. the work is not bounded/fixed but it increases).

IDEs and higher level tools help engineers become more productive. They can do more, with less. This raises the bar on products, and the demand of customers for them (things are pretier, easier to use, etc..), which in turn creates more domains for software to be used, and more demand for engineers.

Google "Induced Demand"

Re: Yann LeCun on GPT-3

#199

Post by maxlamb »

leftyted wrote: post #18

Reading this is really interesting: > GPT-3 doesn't have any knowledge of how the world actually works. I think this is a philosophical question. There is a view that, basically, there is no such thing as knowledge, just language (or, at least, there is no distinction between knowledge and language). In this view, all there really is is language, which is mostly composed of metaphors and, ultimately, metaphors only r…

Humans are very good at visual memory, thanks to millions of years of evolutionary pressure to hunt game and gather for food. Hence why the best memory champions in the world mostly use the concept of "memory palaces." When you think of a car, do you see a car in your head or does a dictionary definition (i.e. purely language knowledge) of a car pop in your mind?

Re: Yann LeCun on GPT-3

#200

Post by babesh »

leftyted wrote: post #18

Reading this is really interesting: > GPT-3 doesn't have any knowledge of how the world actually works. I think this is a philosophical question. There is a view that, basically, there is no such thing as knowledge, just language (or, at least, there is no distinction between knowledge and language). In this view, all there really is is language, which is mostly composed of metaphors and, ultimately, metaphors only r…

Tell that to all the other animals on Earth. Do they not also have knowledge? Do you really think they encode their knowledge in language?

Do you really think that humans are so special as to encode all their knowledge in language? Watch a movie. Listen to a song. Examine a piece of art. Feel sculpture. Play a guitar. Dance.

There is a segment of the software community that is highly language centric/adept. But that community is often blind to other forms of understanding.

Just look at the language of Shakespeare. Much of the language is visual and experiential. How much would you actually understand without your senses and imagination? Your knowledge encompasses your being.

Post reply on HN