Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Yann LeCun on GPT-3

facebook.com

Post reply on HN
Print view View original

121–130 of 253 posts

Re: Yann LeCun on GPT-3

#121

Post by FartyMcFarter »

msamwald wrote: post #107

The original Nabla article is missing information on how they primed GPT-3 for each use-case, and how much effort they put into finding good ways of priming. All fancy GPT-3 demos seem to rely on good priming. The time scheduling problems are probably hard limit of GPT-3 capabilities. The "kill yourself" advice, on the other hand, might have been avoided by better priming.

Wouldn't this kind of priming be brittle and unreliable? Has anyone successfully primed GPT-3 to solve complex problems consistently?

Re: Yann LeCun on GPT-3

#122

Post by dwaltrip »

sailingparrot wrote: post #110
DavidSJ wrote: post #95

Earlier quoted context omitted.

From a business perspective, this is an irrelevant distinction. The requirement was satisfied in a different way, i.e. the engineers satisfying the requirement were replaced by GPT-3, the tool which satisfies the requirement. I think everyone understood that.

The thread is not about business perspective, it's about the hype around what GPT-3 is and is not able to do. One thing GPT-3 is not able to do for example, is replacing 2 ML engineers to build a GPT-3 like model. But OpenAI can do that.

It’s not clear what you are trying to say.

Re: Yann LeCun on GPT-3

#123

Post by 6gvONxR4sf7o »

st1x7 wrote: post #2

It's nice to hear from someone who knows what they're talking about that GPT-3 is just a fancy and expensive autocomplete. The hype in some circles about it went as far as comparing it to AGI at some point which is just ridiculous.

There’s a lot of baggage being thrown into the word fancy here. Any (and I mean any) distribution can be factored as a sequence of its random variables, with the next one being conditional on everything that’s come before, aka autocomplete.

That said, I agree more closely with LeCun than the hypers here.

Re: Yann LeCun on GPT-3

#124

Post by fallous »

disambiguation wrote: post #80
leftyted wrote: post #18

Reading this is really interesting: > GPT-3 doesn't have any knowledge of how the world actually works. I think this is a philosophical question. There is a view that, basically, there is no such thing as knowledge, just language (or, at least, there is no distinction between knowledge and language). In this view, all there really is is language, which is mostly composed of metaphors and, ultimately, metaphors only r…

I don't know if i'd go as far as to agree that "there is no knowledge, only language" .. but I 100% agree one of the key insights from GPT-3 -- why training on language is so effective in the first place -- is that language is tightly coupled to reality

I'm not sure that you can assert that language is tightly coupled to reality, unless you're using the term "reality" to mean something akin to "as one perceives the world" (regardless of whether that perception is correct or not).

Most expressions of language that survived from a few thousand years ago are centered around myths, and while those myths may have contained certain moral or ethical lessons (that were and are subject to interpretation) they certainly weren't tightly coupled to reality in an objective sense.

Training on expressions of language (I separate the concept of language itself from its expression in the form of writing, speaking, etc) certainly has use cases but can GPT-3 recognize a previously unknown analogy and correlate it with the proper piece of applicable "knowledge" it has? If not then it really has no understanding.

Re: Yann LeCun on GPT-3

#125

Post by dwaltrip »

btrask wrote: post #20
drcode wrote: post #11

Earlier quoted context omitted.

What evidence do I have that I'm more than a fancy autocomplete, myself? The use of squishy protestations, in lieu of objective metrics, make LeCun's argument rather unconvincing.

If you are nothing more than fancy autocomplete, then why should we argue with you? GPT is not trying to make a point and is not capable of changing its mind. You, hopefully, are. Edit: I don't think you should be getting downvoted because it's a valid (and interesting) question.

That is an interesting point. However, one can potentially conceive of us human changing our minds as a form of real-time model updating.

Re: Yann LeCun on GPT-3

#127

Post by 6gvONxR4sf7o »

What I think he misses is that with a massive corpus and top tier specialist researchers, sure you can definitely do better, but the point of a plain-text-programmed few shot learner as a product is that it’s better than your average startup’s ML team can confidently produce. If nothing else then because of the training money dumped into it.

Jury’s out on whether the things it’s better at matter much in the marketplace. If I want to know George Washington’s birthday I’ll ask google.

Re: Yann LeCun on GPT-3

#129

Post by dunefox »

InfiniteRand wrote: post #34

One thing I've been wondering, could you train a GPT-3 model to generate "better" text prompts for another GPT-3 model By better I mean grading based on whether there is any nonsense in the output or any internal contradictions, or similar criteria

> By better I mean grading based on whether there is any nonsense in the output or any internal contradictions, or similar criteria

Sounds like you want a hard ai to determine whether a language model generates nonsense.

Re: Yann LeCun on GPT-3

#130

Post by monktastic1 »

tiborsaas wrote: post #55
st1x7 wrote: post #2

It's nice to hear from someone who knows what they're talking about that GPT-3 is just a fancy and expensive autocomplete. The hype in some circles about it went as far as comparing it to AGI at some point which is just ridiculous.

You are just a fancy and efficient autocomplete too. When you speak or write, some words have a higher probability than others. You pick alternatives, but they are limited. Of course there are more layers in the human mind, but GPT-3 is a really impressive milestone towards AGI. It's so easy to downplay every advanced tech, it's actually fun. Planes? Just a flying metal tube. Self landing rockets? Just applied physic…

> You are just a fancy and efficient autocomplete too.

We don't know enough about human cognition to say this.

Scott Aaronson has something interesting to say about this in a conversation with Lex Fridman, actually: https://youtu.be/G_-BBniFFCM?t=419

Quick copy and paste of part of the transcript:

> Humans have a lot of predictive processing a lot of just filling in the blanks but we also have these other mechanisms that we can couple to or that we can sort of call the subroutines when we need to and that maybe maybe you know to go further that one would want to integrate other forms of reasoning.

Post reply on HN