The Bitter Lesson (2019) [pdf]
cs.utexas.edu
The Bitter Lesson (2019) [pdf]
1–10 of 62 posts
Re: The Bitter Lesson (2019) [pdf]
#2This is the original page, I’d link to this:
http://www.incompleteideas.net/IncIdeas/BitterLesson.html
u/dang, swap links if you see this?
Re: The Bitter Lesson (2019) [pdf]
#3On HN before, at least once.[1]
Re: The Bitter Lesson (2019) [pdf]
#4Just adding an obvious corollary regarding parallelization, to "leverage computation" and "massive computation became available and a means was found to put it to good use":
Parallelizable methods win (e.g. DL), by using the computation available
Re: The Bitter Lesson (2019) [pdf]
#5Just adding an obvious corollary regarding parallelization, to "leverage computation" and "massive computation became available and a means was found to put it to good use": Parallelizable methods win (e.g. DL), by using the computation available Further: progress will occur on real-world problems that can be solved by those methods, making those applications dominant throughout society. (i.e. everything looks like a…
Yes. Prior to this though in the AI community there was a lot of debate about the pros and cons of "weak search" (ie generalized methods) and "strong search" (ie methods which make use of domain knowledge). The bitter lesson is more or less that in the long run weak search always wins and the only real judgement call is whether strong search can give you a transient benefit in a particular case before weak search surpasses it and whether that is worth it.
In my head this always parallelled the "premature optimization" conversation in programming. Most programmers would say that inlining etc was only justified when you've benchmarked so you know how the benchmarks are performing but the experience of whole program optimization in things like hotspot jvm and llvm seems to suggest that even benchmarked optimizations can be premature because only a vm can do optimizations on the real world use case.
Re: The Bitter Lesson (2019) [pdf]
#6Also see A Better Lesson:
https://rodneybrooks.com/a-better-lesson/
Re: The Bitter Lesson (2019) [pdf]
#7I think the Go example is facetious. For the longest time, Go simply was not amenable to search. It is only because of breakthrough in theory that Go became amendable to search. And, it is hardly a straightforward search mind you. Two self learning networks are involved. A first to guide the expansion of a Monte Carlo based tree search, a second to evaluate the nodes in the tree. You can hardly blame a researcher in the 70s, 80s, 90s, 00s for not focusing on computer power, when you have 19^2 expansion factor for each move.
Re: The Bitter Lesson (2019) [pdf]
#8I think the Bitter Lessons do not consider two important points with regards to neural nets :
1. Not all nets work for all problems, those that work tend to have the right inductive biases. We discovered the architectures partially by trial an error, nevertheless they work because of encoded prior information.
2. Data and computation are bounded. GPT4 was basically trained on all text, further advancements probably need more insight not more data.
Re: The Bitter Lesson (2019) [pdf]
#9I think the Bitter Lessons do not consider two important points with regards to neural nets : 1. Not all nets work for all problems, those that work tend to have the right inductive biases. We discovered the architectures partially by trial an error, nevertheless they work because of encoded prior information. 2. Data and computation are bounded. GPT4 was basically trained on all text, further advancements probably n…
What do you mean with more insights? If they are not represented as some data, how could they ever influence any computational model?
Re: The Bitter Lesson (2019) [pdf]
#10I think the Go example is facetious. For the longest time, Go simply was not amenable to search. It is only because of breakthrough in theory that Go became amendable to search. And, it is hardly a straightforward search mind you. Two self learning networks are involved. A first to guide the expansion of a Monte Carlo based tree search, a second to evaluate the nodes in the tree. You can hardly blame a researcher in…
It's not called "the bitter lesson" for being optimistic.
One of the points, I think, is that there is very little a researcher in the 70s could have done to make progress on these problems, because the computing power just wasn't there yet; and once the computing power was here, almost all of that 70s researcher's work became obsolete.