Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Bitter Lesson (2019)

incompleteideas.net

Post reply on HN
Print view View original

81–87 of 87 posts

Re: The Bitter Lesson (2019)

#81

Post by sqrt17 »

visarga wrote: post #24
sqrt17 wrote: post #16

Earlier quoted context omitted.

Here's a thing: incorrect assumptions that are built into a model are more harmful than a model that assumes too little structure. If you model the vocal tract and the actual exciting things are the transient noises that occur when we produce consonants, at best there's lots of work with not much to show and at worst you're limiting your model in a negative way. That's the basis for the "every time we fired a linguis…

I think we're far from having used all the media on the internet to train a model. GPT-3 used about 570GB of text (about 50M articles). ImageNet is just 1.5M photos. It's still expensive to ingest the whole YouTube, Google Search and Google Photos in a single model. And the nice thing about these large models is that you can reuse them with little fine-tuning for all sorts of other tasks. So the industry and any hack…

GPT-3 is too large to be useful for practical purposes. Look it up. It's the equivalent of a Formula 1 car or a Saturn V rocket - an impressive feat of technology but of no practical relevance for getting you to work and back.

And certainly fine-tuning and distillation are part of the story why we wanted these large do-all-be-all models in the first place, but the question of what's next for the state of the art - and that currently would be featurization through a large transformer model (i.e. BERT, ERNIE, GPT-2) with some deep-but-not-huge task-specific model on top - isn't simply answered by "more compute".

Re: The Bitter Lesson (2019)

#82

Post by StevenWaterman »

ksdale wrote: post #47

I think it's plausible that many technological advances follow a similar. Something like the steam engine is a step-improvement, but many of the subsequent improvements are basically the obvious next step, implemented once steel is strong enough, or machining precise enough, or fuel is refined enough. How many times has the world changed qualitatively, simply in the pursuit of making things quantitatively bigger or f…

I agree, the first AGI probably will be bodget together with loads of expert input. However, that's not evidence against the bitter lesson.

The first of anything is usually made with the help of experts, but they're quickly overtaken by general methods that lever additional computation

Re: The Bitter Lesson (2019)

#84

Post by YeGoblynQueenne »

acidbaseextract wrote: post #74
YeGoblynQueenne wrote: post #59

Earlier quoted context omitted.

>> Why do you dismiss the unconscious search that humans do in Go? The question is why you say that humans perform an unconscious search when they play Go. And what kind of search is it, other than unconscious? Could you describe it, e.g. in algorithmic notation? I mean, I'm sure you couldn't because if you could then the problem of teaching a computer to play Go as well as a human would have been solved years and ye…

Here's a section of Michael Redmond's (9-dan professional Go player) commentary on the Lee Sedol vs AlphaGo matches: https://youtu.be/yCALyQRN3hw?t=3031 It's really fun to watch his commentary because he relentlessly plays "variations" — possible next moves and sequences — while waiting for the players, explaining the tradeoffs between moves and the consequences they lead to a few steps ahead in the game. I don't kno…

Fast enough to evaluate a few million future positions in a few seconds? Like I say in another comment, even professional players cannot "look ahead" more than a few ply, so whatever it is they're doing "in their heads", the tree search they're reporting is not how they win games.

To clarify, you can come up with an explanation of anything that you do, or observe yourself or another person do. For example, you might explain how you hit a ball with racket in tennis or with a bat in baseball, etc, but that doesn't mean that the process you are describing is the process that your mind (let alone your brain) actually follows.

If nothing else because such a description will necessarily fudge important steps. For example, if I describe myself walking as "I put one foot in front of the other" - have I explained enough about walking that it can now be reproduced mechanically? Experience teaches that -no.

Re: The Bitter Lesson (2019)

#85

Post by ksdale »

StevenWaterman wrote: post #82
ksdale wrote: post #47

I think it's plausible that many technological advances follow a similar. Something like the steam engine is a step-improvement, but many of the subsequent improvements are basically the obvious next step, implemented once steel is strong enough, or machining precise enough, or fuel is refined enough. How many times has the world changed qualitatively, simply in the pursuit of making things quantitatively bigger or f…

I agree, the first AGI probably will be bodget together with loads of expert input. However, that's not evidence against the bitter lesson. The first of anything is usually made with the help of experts, but they're quickly overtaken by general methods that lever additional computation

Sorry, I didn't mean to suggest that the bitter lesson is wrong, just that it's not bitter, it's actually how a whole bunch of stuff progresses.

Re: The Bitter Lesson (2019)

#86

Post by StevenWaterman »

ksdale wrote: post #85
StevenWaterman wrote: post #82

Earlier quoted context omitted.

I agree, the first AGI probably will be bodget together with loads of expert input. However, that's not evidence against the bitter lesson. The first of anything is usually made with the help of experts, but they're quickly overtaken by general methods that lever additional computation

Sorry, I didn't mean to suggest that the bitter lesson is wrong , just that it's not bitter, it's actually how a whole bunch of stuff progresses.

Makes sense!

Re: The Bitter Lesson (2019)

#87

Post by francoisp »

YeGoblynQueenne wrote: post #73
francoisp wrote: post #36

building a model for and with domain knowledge == premature optimization? In the end a win on kaggle or a published paper seems to depend on tweaking hyperparameters based on even more pointed DK: data set knowledge... I wonder what would be required to build a model that explores the search space of compilable programs in say python that sorts in correct order. Applying this idea of using ML techniques to finding be…

>> I wonder what would be required to build a model that explores the search space of compilable programs in say python that sorts in correct order. Oh, not that much. You could do that easily with a small computer and an infinite amount of time.

With Deep Thought and an infinite amount of time, you can get the answer to life the universe and everything...
Post reply on HN