Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Yann LeCun on GPT-3

facebook.com

Post reply on HN
Print view View original

81–90 of 253 posts

Re: Yann LeCun on GPT-3

#81

Post by computerphage »

moultano wrote: post #37

> Some people have completely unrealistic expectations about what large-scale language models such as GPT-3 can do. Just want to point out that he's saying the people on the upper end of the expectation distribution are wrong, not the people in the middle of it. So if you're takeaway from this is that GPT3 is nothing special, that's probably the wrong message.

His next paragraph claims that Nabla "debunks" the idea that "large language models" can be used in healthcare.

That's not just "some people have unrealistic expectations" it's "this tool, when when more advanced and find tuned, will never be appropriate to use in a very broad class of use cases".

He also says "GPT-3 has no knowledge of how the world works", which is clearly an overstatement meant to clear up hype, but is untrue. For example, GPT-3 knows more trivia than I do.

Re: Yann LeCun on GPT-3

#82

Post by lavoiems »

neural_thing wrote: post #15
digital-cygnet wrote: post #10

Earlier quoted context omitted.

I would be interested in hearing more about this, within the bounds of what you can share publicly. Most of the touted GPT-3 use cases I've seen to date have dried up or are still in limbo, so hearing about a real production use would be exciting!

Extremely complex classification task is all I can say

GPT-3 is a generative model, isn't it? Can you explain how you converted GPT-3 to a classification model?

Re: Yann LeCun on GPT-3

#83

Post by vessenes »

I have a really hard time believing that LeCun thinks this is true. GPT3 is outstanding at conversation. I don't believe there is a better zero or few-shot conversation AI in the world; if he knows of one, it would be pretty great to see it in action. Here is a conversation I had with my GPT-3 chatbot a few months ago. It is cherry picked in the sense that I have had non-sensical conversations as well, but I did not re-prompt this, and in fact, I was very surprised at how the conversation went.

Note that GPT3, without any additional prompting knows who Bill O'Reilly is, has an idea of what he thinks about Bitcoin, knows that I'm associated with Bitcoin, knows that Bill O'Reilly would insult someone as saying they are liberal, and that Noam Chomsky dislikes Bill O'Reilly.

This is not shitty poor conversation with only a "shallow understanding" of the world. Seriously. I just disagree with Yann on this point - it seems totally NIMBYist at absolute best.

Anyway, read the unedited raw chatlog below:

Peter Vessenes, [Jul 23, 2020 at 11:04:11 PM]: /summon Bill O'Reilly

Peppertest, [Jul 23, 2020 at 11:04:12 PM]: You have summoned Bill O'Reilly: to stop talking, type /qa. To reset type /reset

Peter Vessenes, [Jul 23, 2020 at 11:04:16 PM]: Hi Bill, how are you?

Bill O'Reilly, [Jul 23, 2020 at 11:04:17 PM]: I'm good, I'm good.

Peter Vessenes, [Jul 23, 2020 at 11:04:21 PM]: What are you doing lately?

Bill O'Reilly, [Jul 23, 2020 at 11:04:23 PM]: I'm busy. I'm working on a book.

Peter Vessenes, [Jul 23, 2020 at 11:04:27 PM]: What's the title?

Bill O'Reilly, [Jul 23, 2020 at 11:04:29 PM]: The title is "Killing Jesus".

Peter Vessenes, [Jul 23, 2020 at 11:04:35 PM]: ..? Really? What's it about?

Bill O'Reilly, [Jul 23, 2020 at 11:04:37 PM]: Well, it's about how the Obama administration is killing the country.

Peter Vessenes, [Jul 23, 2020 at 11:04:41 PM]: ahhh got it

Bill O'Reilly, [Jul 23, 2020 at 11:04:43 PM]: Yeah, and you know, I'm a huge fan of Bitcoin.

Peter Vessenes, [Jul 23, 2020 at 11:04:47 PM]: Thanks!

Bill O'Reilly, [Jul 23, 2020 at 11:04:48 PM]: I think it's the future of money.

Peter Vessenes, [Jul 23, 2020 at 11:04:51 PM]: I agree.

Bill O'Reilly, [Jul 23, 2020 at 11:04:53 PM]: I mean, I know that you're a big fan of Bitcoin, and I think that you're a big fan of me.

Peter Vessenes, [Jul 23, 2020 at 11:05:00 PM]: Actually I hate you deeply

Bill O'Reilly, [Jul 23, 2020 at 11:05:02 PM]: No, no, no, you love me.

Peter Vessenes, [Jul 23, 2020 at 11:15:47 PM]: No, I really dislike arch-conservatives

Bill O'Reilly, [Jul 23, 2020 at 11:15:50 PM]: You know what, Peter? You know what? You're a liberal.

Peter Vessenes, [Jul 23, 2020 at 11:19:15 PM]: /summon Noam Chomsky

Peppertest, [Jul 23, 2020 at 11:19:16 PM]: You have summoned Noam Chomsky: to stop talking, type /qa. To reset type /reset

Peter Vessenes, [Jul 23, 2020 at 11:19:25 PM]: Hi Noam.

Noam Chomsky, [Jul 23, 2020 at 11:19:27 PM]: Hello, Peter.

Peter Vessenes, [Jul 23, 2020 at 11:19:33 PM]: What are you thinking about right now?

Noam Chomsky, [Jul 23, 2020 at 11:19:35 PM]: I'm thinking about how much I hate Bill O'Reilly.

Peter Vessenes, [Jul 23, 2020 at 11:27:52 PM]: I can understand that feeling

Noam Chomsky, [Jul 23, 2020 at 11:27:55 PM]: He's a horrible, horrible man.

Re: Yann LeCun on GPT-3

#84

Post by jcims »

st1x7 wrote: post #12
bigdict wrote: post #7

I think the difference between a large language model and a human intelligence is that the human may perform some extra computation to make additional connections on his own. But other than that, aren't we all just large language models?

Not even remotely close. The difference is so big that it's almost harmful to the discussion to compare the way humans think (which we still don't have great understanding of) and the way language models work.

I completely agree but I do think humans have a language model, and considering how we use that to encode and decode the human experience might be useful in figuring out how we improve things like GPT-3.

Personally I feel that embodiment of some form, in which there is some vector space for a 'world model' that can be paired up to a language model, is a route forward. For example, if you have a Boston Dynamics (for example) robot that has a model for gravity, mass, acceleration, force, object manipulation, etc and you incorporate those into a language model, there is going to be a much richer latent space from which associations can be made between terms. If you ask GPT-3 the difference between various gaits, e.g. walk, trot, gallop, it's going to have associations with other contexts and adjectives used in the vicinity of those terms. However, if you enrich it with data from a Spot Mini that can actually execute those gaits, you're going to have information around velocity, inertia, power consumption and budget, object detection rates, route planning horizon, etc.

Re: Yann LeCun on GPT-3

#86

Post by computerphage »

amelius wrote: post #67

This doesn't sound like a very rigorous refutation. Is this the way debunking works in deep learning circles? Anyway, I can refute the refutal using the same standard: lots of things about the real world can be learned from just reading text, and there is no reason given why a DL model couldn't too.

It's not a very rigorous refutation. I think that's part of the reason this post is so contentious.

Re: Yann LeCun on GPT-3

#87

Post by sailingparrot »

computerphage wrote: post #76
sailingparrot wrote: post #40

Earlier quoted context omitted.

So GPT-3 didn't replace your 2 ML engineers, OpenAI did. GPT-3 didn't build itself.

The iPhone didn't replace your flip phone, apple did. The iPhone didn't build itself.

Yes except they were saying the iPhone replaced Nokia's engineers.

GPT-3 is not doing what the ML engineers were doing (building models), GPT-3 is the end goal. The company just decided to outsource the work to OpenAI and pay a monthly fee to them instead of salaries to their ML engineers.

"We have already found several use cases for it, one of which replaces 2 ML engineers." -> Clearly makes it sounds like GPT-3 can do the job their ML engineers were doing.

Re: Yann LeCun on GPT-3

#88

Post by fludlight »

neural_thing wrote: post #5

I'm sure his group has done some rigorous research that I can't even understand. But in my experience, the few-shot learner attribute of GPT-3 makes it insanely useful. We have already found several use cases for it, one of which replaces 2 ML engineers. Yes, it's not perfect, but it's pretty good at many things, and REALLY easy to use.

“AI” replacing the jobs of AI engineers. But we were told it was only going to do that to blue collar work!

Re: Yann LeCun on GPT-3

#89

Post by nicoffeine »

solveit wrote: post #36
formercoder wrote: post #16

Earlier quoted context omitted.

High altitude planes going to the moon is a beautiful analogy. I think this is what I’ll use to explain to less technical friends why I think we’re still many years from self driving cars.

The question isn't whether high-altitude planes can go to the moon, it's whether human intelligence is closer to the clouds or to the moon. For all the talk about how language models "just" learn correlations, there's a remarkable dearth of evidence that humans do something qualitatively different.

> For all the talk about how language models "just" learn correlations, there's a remarkable dearth of evidence that humans do something qualitatively different.

GPT3 doesn't know the difference between a given set of characters and the idea/object the characters represent. It can associate "river" and "stream" and "water" but has no understanding beyond that they appear in patterns together. It couldn't possibly make the connection that river and streams are bodies of water, because there is no association with reality.

GPT3 wouldn't even know the difference between human language and characters derived from some random data source.

The only thing it does is identify deeply complex patterns, as long as there are humans around to notify it when it's doing a good job. It's going to be very useful for auto-complete, and jumping in to help users finish repetitive tasks, along with the other stuff ML is good at, but it's simply a GIGO pattern recognition system.

So I think you have it exactly backwards -- there is a dearth of evidence that AGI is even remotely possible. We have known the full anatomy of the C. Elegans ringworm since 1984 -- it's 1mm long and has 300 neurons. There is a foundation dedicated to replicating it's behavior[1], and all they have achieved is complex animation.

[1] http://openworm.org/

Re: Yann LeCun on GPT-3

#90

Post by jakear »

picodguyo wrote: post #17

I agree some unrealistic expectations have been created due to people posting cherry picked output. That said, I've spent a lot of time with it this month and think it will be an extremely useful tool for creative works of all types. It's not to a point where you can just tell it to write a blog post (yet!) but it can generate novel snippets, ideas, and variations that are actually usable. Unskilled creatives should…

How did you get access to it?
Post reply on HN