Potentially also of interest is Rod Brooks' response "A Better Lesson" (2019): https://rodneybrooks.com/a-better-lesson/
The Bitter Lesson (2019)
21–30 of 87 posts
Re: The Bitter Lesson (2019)
#22This reminds me of the Banko and Brill paper "Scaling to very very large corpora for natural language disambiguation" - https://dl.acm.org/doi/10.3115/1073012.1073017.
It is exactly the point and it is something not a lot of researchers really grok. As a researcher you are so smart, why can't you discover whatever you are seeking? I think in this decade, we see a couple more scientific discoveries by brute force which will hopefully will make the scientific type a bit more humble an honest.
Re: The Bitter Lesson (2019)
#23The article says we should focus on increasing the compute we use in AI instead of embedding domain specific knowledge. OpenAI seems to have taken this lesson to heart. They are training a generic model using more compute than anything else. Many researchers predict a plateau for AI because it is missing the domain specific knowledge but this article and the benefits of more compute that OpenAI is demonstrating beg t…
Model compression is an active research field and will probably be quite lucrative, as you will literally able to save millions.
Re: The Bitter Lesson (2019)
#24It’s funny when you’ve been thinking for months about how speech recognition could really benefit from integrating models of the human vocal tract… and then you read this
Here's a thing: incorrect assumptions that are built into a model are more harmful than a model that assumes too little structure. If you model the vocal tract and the actual exciting things are the transient noises that occur when we produce consonants, at best there's lots of work with not much to show and at worst you're limiting your model in a negative way. That's the basis for the "every time we fired a linguis…
I think we're far from having used all the media on the internet to train a model. GPT-3 used about 570GB of text (about 50M articles). ImageNet is just 1.5M photos. It's still expensive to ingest the whole YouTube, Google Search and Google Photos in a single model.
And the nice thing about these large models is that you can reuse them with little fine-tuning for all sorts of other tasks. So the industry and any hacker can benefit from these uber-models without having to retrain from scratch. Of course, if they even fit the hardware available, otherwise they have to make due with a slightly lower performance.
Re: The Bitter Lesson (2019)
#25Got to believe, this is like heroin. Its a win until it isn't. Then where will AI researchers be? No progress for 20 (50?) years because the temptation to not understand but to just build performant engineering solutions, was so strong. In fact, is the researcher supposed to be building the most performant solution? This article seems alarmingly misinformed. To understand 'artificial intelligence' isn't a race to VC…
>> This article seems alarmingly misinformed. I hate appeals to authority as much as anybody else on HN, but I'm not sure that we could say Rich Sutton[1] is "misinformed". He's an established expert in the field, and if we discount his academic credentials then at least consider he's understandably biased towards this line of thinking as one of the early pioneers of reinforcement learning techniques[2] and currently…
He's writing that article for a reason, to be sure. Its just not the one that the article says its about, I'm thinking.
Re: The Bitter Lesson (2019)
#26Got to believe, this is like heroin. Its a win until it isn't. Then where will AI researchers be? No progress for 20 (50?) years because the temptation to not understand but to just build performant engineering solutions, was so strong. In fact, is the researcher supposed to be building the most performant solution? This article seems alarmingly misinformed. To understand 'artificial intelligence' isn't a race to VC…
AI as a field relied mostly on 'understanding' based approaches for 50 years without much success. These approaches were too brittle and ungrounded. Why return to something that doesn't work? DNNs today can generate images that are hard to distinguish from real photos, super natural voices and surprisingly good text. They can beat us at all board games and most video games. They can write music and poetry better than…
Because, they can't invent a new game. Unless of course they were only designed to invent games, and by trial and error and statistical correlation to existing games, thus producing a generic thing that relates to everything but invents nothing.
I'm not an idiot. I understand that we won't have general purpose thinking machines any time soon. But to give up entirely looking into that kind of thing, seems to me to be a mistake. To rebrand the entire field as calculating results to given problems and behaviors using existing mathematical tools, seems to do a disservice to the entire concept and future of artificial intelligence.
Imagine if the field of mathematics were stumped for a while, so investigators decided to just add up things faster and faster, and call that Mathematics.
Re: The Bitter Lesson (2019)
#27I have read this before and broadly agree with the point—it’s no use trying to curate expertise into AI. But I don’t think modeling p(y|x) or it’s friend p(y, x) is the end we’re looking for either. But, it’s unreasonably effective, so we keep doing it. (I don’t have an answer or an alternative; causality appeals to my intuition, but it’s really clunky and has seemingly not paid off.)
Actually I feel like causalities time has come. The framework that has convinced me is just the simple approach of doing controlled experiments over observational data to establish causal links via DAGs no need for any drama!
It seems to be just shuffling around the hard part of the problem. Causality still depends on some unstructured optimization problem of generating and evaluating causal diagram candidates. I haven’t really seen it applied where the set of potential causal relationships is huge.
Re: The Bitter Lesson (2019)
#28Earlier quoted context omitted.
AI as a field relied mostly on 'understanding' based approaches for 50 years without much success. These approaches were too brittle and ungrounded. Why return to something that doesn't work? DNNs today can generate images that are hard to distinguish from real photos, super natural voices and surprisingly good text. They can beat us at all board games and most video games. They can write music and poetry better than…
Because, they can't invent a new game. Unless of course they were only designed to invent games, and by trial and error and statistical correlation to existing games, thus producing a generic thing that relates to everything but invents nothing. I'm not an idiot. I understand that we won't have general purpose thinking machines any time soon. But to give up entirely looking into that kind of thing, seems to me to be…
What GPT-3 and other models lack is embodiment. There are of course RL agents embodied in simulated environments, like games and robot sims, but this pales in comparison to our access to nature and the human society. When we will be able to give them a body they will naturally rediscover play and games.
Human superiority doesn't come just from the brain, it comes from the environment this brain has access to - other humans, culture, tools, nature, and the bodily affordances (hands, feet, eyes, ability to assimilate organic food...). AI needs a body and an environment to evolve in.
Re: The Bitter Lesson (2019)
#29Interesting, I wonder what happens now that Moore's law is considered dead and we can't rely on computation power increasing year over year. To make further progess with general purpose search and learning methods we will need lots more computational power which may not be cheaply available. Then do we focus our efforts on developing more efficient learning strategies like the one we have in our minds ? I do agree wi…
it's kind of poetic that the chief bottleneck of advancement in the field is now the physical universe -
Re: The Bitter Lesson (2019)
#30Interesting, I wonder what happens now that Moore's law is considered dead and we can't rely on computation power increasing year over year. To make further progess with general purpose search and learning methods we will need lots more computational power which may not be cheaply available. Then do we focus our efforts on developing more efficient learning strategies like the one we have in our minds ? I do agree wi…
Can you elaborate why you think that Moore's law is considered dead? For me it seems that the general progress for the computing hardware in question (GPUs and specialized ASICs, not consumer CPUs) we're still seeing steady improvements in transistors/$ and flops/$ and expect it to still continue for some time at least.