Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Yann LeCun on GPT-3

facebook.com

Post reply on HN
Print view View original

111–120 of 253 posts

Re: Yann LeCun on GPT-3

#111

Post by vladf »

tmalsburg2 wrote: post #42
vladf wrote: post #21

Earlier quoted context omitted.

I happen to use slightly less fancy and expensive GPT-2 based autocomplete, and it's amazing. https://tabnine.com

Interesting. As a reading researcher, I imagine that this could potentially introduce subtle and difficult to spot bugs when you get a proposed completion that looks about right i.e. close enough to what you imagined. Has this been an issue in your experience?

No, it's not auto-completing function blocks, just simple expressions that are easy to validate. E.g.,

    let lo = 0;
    let hi = vec.len();
    let mid = lo + (hi
will autocomplete to `(hi - lo) / 2` as the second autocomplete option (so I'd hit tab twice). If you were to "score" it based on top-5 it'd probably be pretty bad at guessing my intent, but then again, I get to _opt-in_ to suggestions so it just needs to be right often enough, as it doesn't bother me much to keep typing.

Re: Yann LeCun on GPT-3

#112

Post by sooheon »

this_was_posted wrote: post #66

To me GPT-3 feels more like a rocket-booster than a high-altitude plane. On its own it's not going to reach the moon, but combined with the right guidance and additional thrust it just might. I think being able to model future outcome of something in a similar way humans would (like GPT-3 does) is the first input step for an overarching AI to reach some kind of sentience. With my admittedly limited understanding I be…

Stack more GPT-3s! Have GPT-ception via stacks of multi-headed GPT blocks. I'm sure softmax attention can be modeled as a few-shot text generation problem.

Re: Yann LeCun on GPT-3

#113

Post by yorak »

sailingparrot wrote: post #98
computerphage wrote: post #81

Earlier quoted context omitted.

His next paragraph claims that Nabla "debunks" the idea that "large language models" can be used in healthcare. That's not just "some people have unrealistic expectations" it's "this tool, when when more advanced and find tuned, will never be appropriate to use in a very broad class of use cases". He also says "GPT-3 has no knowledge of how the world works", which is clearly an overstatement meant to clear up hype, b…

> which is clearly an overstatement meant to clear up hype, but is untrue It all depends on your definition of knowledge. Under a certain definition you could say that GPT-3 knows basically nothing. If someone teaches me to repeat perfectly something very smart in a language I don't know, without explaining to me what that thing is, do I have knowledge about this? The same argument can be made about those kind of mod…

Aka. the Chinese room argument. However, I'm not so sure us people are little more than just pattern matching machines. When I start to talk (or write, as I'm doing now), the words kind of just flow out. I can make the argument, that I understand the "real" world, but do I really?

Re: Yann LeCun on GPT-3

#114

Post by whymauri »

fludlight wrote: post #88
neural_thing wrote: post #5

I'm sure his group has done some rigorous research that I can't even understand. But in my experience, the few-shot learner attribute of GPT-3 makes it insanely useful. We have already found several use cases for it, one of which replaces 2 ML engineers. Yes, it's not perfect, but it's pretty good at many things, and REALLY easy to use.

“AI” replacing the jobs of AI engineers. But we were told it was only going to do that to blue collar work!

Because most "AI engineering" has lost its meaning and is actually data analysis.

Re: Yann LeCun on GPT-3

#115

Post by thomas_rm »

lisper wrote: post #75
thomas_rm wrote: post #49

Earlier quoted context omitted.

To remind people: Yann LeCun worked on artificial neural networks (ANN) during the period where they were actively shunned by most of the scientific community. You could barely publish a paper on ANN. Just to demonstrate, one the most common books during period, "Artificial Intelligence: A Modern Approach, 2nd ed" by Norvig, 1080 pages, has less than one (1!) page dedicated to ANNs. I personally think Norvig is an id…

> Artificial Intelligence: A Modern Approach, 2nd ed Published in 2002. At that point, ANN research had reached a pretty hard plateau with very few tangible results. Faulting Russel and Norvig for not going into depth about ANNs is kind of like faulting Richard Feynman for not going into depth about quantum computers in the Feynman Lectures. Also, a lot of the subsequent work and breakthroughs on ANNs has been done a…

As opposed to the AI techniques taught in the AIMA book (KR and logic reasoning), which had plateau'ed in the 70s...?

Norvig had to be pretty clueless to decide ANNs are such a dead-end, that they don't deserve even a chapter in his book, where all around him there are biological living proofs that neural networks are probably a pretty good bet for AI...

(Note: I held the same opinion in the mid 90s when I reviewed his 1st edition and I'm definitely no Feynmann-level. It's just common sense.)

Re: Yann LeCun on GPT-3

#116

Post by vannevar »

moultano wrote: post #37

> Some people have completely unrealistic expectations about what large-scale language models such as GPT-3 can do. Just want to point out that he's saying the people on the upper end of the expectation distribution are wrong, not the people in the middle of it. So if you're takeaway from this is that GPT3 is nothing special, that's probably the wrong message.

Yes, I think the focus on "getting to the moon,", to use his analogy, ignores the fact that GPT-3 is an SR-71 in a world of 19th century balloons. It may not get to the moon, but it definitely points the way to lots of useful stuff. There is a lot of boilerplate text in the world that is well-suited to auto-generation by a GPT-3-style model. And currently a lot of people employed to generate that text, at significant aggregate expense.

Re: Yann LeCun on GPT-3

#117

Post by whymauri »

andybak wrote: post #92

I've got a friend who tries to talk me down every time i enthuse about GPT-3 or similar. He seems to think I believe it to me more amazing than it is whereas I struggle to convince him that I think I've got a good handle on it's limitations and I still find it mind-bogglingly amazing.

Something I've learned over time is that sometimes it's OK to let people be curious and amazed. The world and science would be far too boring if we were cynical about _all_ of it _all_ the time.

Re: Yann LeCun on GPT-3

#118

Post by picodguyo »

lavoiems wrote: post #82
neural_thing wrote: post #15

Earlier quoted context omitted.

Extremely complex classification task is all I can say

GPT-3 is a generative model, isn't it? Can you explain how you converted GPT-3 to a classification model?

There are a couple of ways to do it. You can give it a prompt that shows examples of the classification and it mimics what it thinks is the correct behavior when you feed it new unclassified input. They also have a search endpoint that lets you do classification by giving it an input along with labels as the searchable documents and using the resulting semantic relevance scores.

Re: Yann LeCun on GPT-3

#119

Post by andrewprock »

katzgrau wrote: post #71
solveit wrote: post #36

Earlier quoted context omitted.

The question isn't whether high-altitude planes can go to the moon, it's whether human intelligence is closer to the clouds or to the moon. For all the talk about how language models "just" learn correlations, there's a remarkable dearth of evidence that humans do something qualitatively different.

> there's a remarkable dearth of evidence that humans do something qualitatively different. Perhaps now, but if history is any indication, when we (as humans) think we have a good grip on how something really works (like human intellect in this example), we've been wrong. We model the world around us from observation and testing, find our errors, remodel, and improve over time. Then at some point we find some piece o…

Sure there is. Just ask a person to engage in a "stream of consciousness" monologue.

Re: Yann LeCun on GPT-3

#120

Post by picodguyo »

jakear wrote: post #90
picodguyo wrote: post #17

I agree some unrealistic expectations have been created due to people posting cherry picked output. That said, I've spent a lot of time with it this month and think it will be an extremely useful tool for creative works of all types. It's not to a point where you can just tell it to write a blog post (yet!) but it can generate novel snippets, ideas, and variations that are actually usable. Unskilled creatives should…

How did you get access to it?

They've opened the spigot on the beta now and thousands of people have access. Just sign up and you should get in fairly quickly.
Post reply on HN