Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Bitter Lesson (2019)

incompleteideas.net

Post reply on HN
Print view View original

11–20 of 87 posts

Re: The Bitter Lesson (2019)

#11

Post by sgt101 »

cgearhart wrote: post #5

I have read this before and broadly agree with the point—it’s no use trying to curate expertise into AI. But I don’t think modeling p(y|x) or it’s friend p(y, x) is the end we’re looking for either. But, it’s unreasonably effective, so we keep doing it. (I don’t have an answer or an alternative; causality appeals to my intuition, but it’s really clunky and has seemingly not paid off.)

Actually I feel like causalities time has come. The framework that has convinced me is just the simple approach of doing controlled experiments over observational data to establish causal links via DAGs no need for any drama!

Re: The Bitter Lesson (2019)

#12

Post by astrophysician »

I think what he's basically saying is that priors (i.e. domain knowledge + custom, domain-inspired models) help when you're data limited or when your data is very biased, but once that's not the case (e.g. we have an infinite supply of voice samples), model capacity is usually all that matters.

Re: The Bitter Lesson (2019)

#13

Post by maest »

For contrast, take this Hofstadter quote:

> This, then, is the trillion-dollar question: Will the approach undergirding AI today—an approach that borrows little from the mind, that’s grounded instead in big data and big engineering—get us to where we want to go? How do you make a search engine that understands if you don’t know how you understand? Perhaps, as Russell and Norvig politely acknowledge in the last chapter of their textbook, in taking its practical turn, AI has become too much like the man who tries to get to the moon by climbing a tree: “One can report steady progress, all the way to the top of the tree.”

My take is that there is something intelectually unsatisfying about solving a problem by simply throwing more computational power at it, instead of trying to understand it better.

Imagine in a parallel universe where computational power is extremely cheap. In this universe, people solve integrals exclusively by numerical integrations so there is no incentive to develop any of the Analysis theory we currently have. I would expect that to be a net negative in the long run as theories like Gen Relativity would be almost impossible to develop without the current mathematical apparatus.

Re: The Bitter Lesson (2019)

#14

Post by sytse »

The article says we should focus on increasing the compute we use in AI instead of embedding domain specific knowledge. OpenAI seems to have taken this lesson to heart. They are training a generic model using more compute than anything else.

Many researchers predict a plateau for AI because it is missing the domain specific knowledge but this article and the benefits of more compute that OpenAI is demonstrating beg to differ.

Re: The Bitter Lesson (2019)

#15

Post by jefft255 »

auggierose wrote: post #9

I guess it depends on what you trying to do. I had a computer vision problem where I was like, hell yeah, let's machine learn the hell out of this. 2 months later, and the results were just not precise enough. It took me 2 more months, and now I am solving the task easily on an iPhone via Apple Metal in milliseconds with a hand-crafted optimisation approach ...

His advice really concerns more scientific research and its long-term progress, and not really immediate applications. I think that injecting human knowledge can lead to faster, more immediate progress, and he seems to believe that too. The "bitter lesson" is that general, data-driven approaches will always win out eventually.

Re: The Bitter Lesson (2019)

#16

Post by sqrt17 »

vlmutolo wrote: post #2

It’s funny when you’ve been thinking for months about how speech recognition could really benefit from integrating models of the human vocal tract… and then you read this

Here's a thing: incorrect assumptions that are built into a model are more harmful than a model that assumes too little structure. If you model the vocal tract and the actual exciting things are the transient noises that occur when we produce consonants, at best there's lots of work with not much to show and at worst you're limiting your model in a negative way. That's the basis for the "every time we fired a linguist, recognition rates improved" from 90s speech recognition.

On the other end of the spectrum, data and compute ARE limited and for some tasks we're at a point where the model eats up all the humanity's written works and a couple million dollars in compute and further progress has to come from elsewhere because even large companies won't spend billions of dollars in compute and humanity will not suddenly write ten times more blog articles.

Re: The Bitter Lesson (2019)

#17

Post by avmich »

> When a simpler, search-based approach with special hardware and software proved vastly more effective, these human-knowledge-based chess researchers were not good losers.

It's like calling Russia a loser in Cold War. Technically the effect is reached; practically the side which "lost" gained possibly largest benefits.

Re: The Bitter Lesson (2019)

#18

Post by otoburb »

JoeAltmaier wrote: post #4

Got to believe, this is like heroin. Its a win until it isn't. Then where will AI researchers be? No progress for 20 (50?) years because the temptation to not understand but to just build performant engineering solutions, was so strong. In fact, is the researcher supposed to be building the most performant solution? This article seems alarmingly misinformed. To understand 'artificial intelligence' isn't a race to VC…

>>This article seems alarmingly misinformed.

I hate appeals to authority as much as anybody else on HN, but I'm not sure that we could say Rich Sutton[1] is "misinformed". He's an established expert in the field, and if we discount his academic credentials then at least consider he's understandably biased towards this line of thinking as one of the early pioneers of reinforcement learning techniques[2] and currently a research scientist at DeepMind leading their office in Alberta, Canada.

[1] https://en.wikipedia.org/wiki/Richard_S._Sutton

[2] http://incompleteideas.net/papers/sutton-88-with-erratum.pdf

Re: The Bitter Lesson (2019)

#19

Post by visarga »

JoeAltmaier wrote: post #4

Got to believe, this is like heroin. Its a win until it isn't. Then where will AI researchers be? No progress for 20 (50?) years because the temptation to not understand but to just build performant engineering solutions, was so strong. In fact, is the researcher supposed to be building the most performant solution? This article seems alarmingly misinformed. To understand 'artificial intelligence' isn't a race to VC…

AI as a field relied mostly on 'understanding' based approaches for 50 years without much success. These approaches were too brittle and ungrounded. Why return to something that doesn't work?

DNNs today can generate images that are hard to distinguish from real photos, super natural voices and surprisingly good text. They can beat us at all board games and most video games. They can write music and poetry better than the average human. Probably also drive better than an average human. Why worry about 'no progress for 50 years' at this point?

Re: The Bitter Lesson (2019)

#20

Post by aszen »

Interesting, I wonder what happens now that Moore's law is considered dead and we can't rely on computation power increasing year over year. To make further progess with general purpose search and learning methods we will need lots more computational power which may not be cheaply available. Then do we focus our efforts on developing more efficient learning strategies like the one we have in our minds ?

I do agree with the part about not embedding human knowledge into our computer models, any knowledge worth learning about any domain the computer should be able learn on its own to make true progress in AI.

Post reply on HN