The Bitter Lesson
incompleteideas.net
The Bitter Lesson
1–10 of 54 posts
Re: The Bitter Lesson
#2This really needs a better title, or at least a subtitle. (I clicked only because I recognized the domain name; the title itself is vague puffery which gives no promise of being interesting...)
'compute beats clever'? 'fast > fancy'? 'better big than bright'? 'in the end, brute force wins'?
Or at least call it "AI's Bitter Lesson" or something!
Re: The Bitter Lesson
#3> Time spent on one is time not spent on the other.
But from the AI researcher's view, "the other" doesn't require time; someone else is advancing the hardware, which is outside of the AI researcher's area. The general method to-be-run on better hardware is known today; it doesn't have to be researched. So should the AI researcher just twiddle their thumbs, waiting for the hardware to improve?
In games like chess, it has long been known that if you have a big enough database, an optimal game can be played. For each board configuration, the entry in the database supplies the optimal move.
So according to the reasoning in this article, we shouldn't bother looking for improving chess. The hardware will eventually get enough storage that we can store all the moves, and fast enough that we can compute them all to populate the table. Then twenty lines of Javascript doing hashed lookup of chess boards with a giant Redis back end will be the grandmaster. :)
Re: The Bitter Lesson
#4It's not just AI. Our trade's history is chockfull of valiant efforts to solve problems that were overrun by the exponential decline of computing costs by the time they really worked properly.
Remember DSEE / ClearCase? They had all sorts of complicated virtual file systems to deliver tagged and branched contents of source code repositories. But drive space expanded with a Moore's-law style curve and now we have "git pull". Far simpler. System administrators don't hate our guts for adopting git.
Remember PHIGS? We needed display lists for graphics engines because host machines were too slow. Silicon Graphics took the other approach, and now we have GL.
Remember terminal concentrators like Digital's LAT? You don't? Good. I wish I didn't. (Handling 9600 baud interrupts was too big a load for a host machine. Really.)
Remember optical typesetting machines? The digital outlines / images for creating nice-looking letters used to be too big and complex to use for creating the images for actual pages. You want to use Univers or Gill Sans to set a document? Fine. Buy a Selectric typeball. Or go pay Linotype or Monotype a bundle for a little optical thingy with images of all the letters on it. Take the lid off your typesetting machine and put that thingy into it. You want to set Japanese? Too bad for you. Apple, Adobe, Chuck Bigelow and Kris Holmes, and Matt Carter, and Donald Knuth, decided to ride the exponential rocket and the rest is history.
The bright side: Margaret Hamilton and her team on the Apollo moonshot project used simple, reliable, radiation-hardened, and redundant computers with bugfree software to get those guys to the moon and back.
Let's be careful: Generals always fight the last war. Before starting new valiant efforts, we should carefully assess whether the appropriate technology for the planned delivery date is JMOS -- Just a Matter of Software. Sometimes it might not be the case. But most often it will be.
Re: The Bitter Lesson
#5The majority of businesses and governments are insisting on learning this bitter lesson anew.
In the minds of many business executives and government officials, "explainable AI" means, quite literally, "show it to me as a linear combination of a small number of features" (sometimes called "drivers" or "factors") that have monotonic relationships with measurable outcomes.
I would go further: most people are understandably scared and worried of intelligence that arises from scalable search and learning by self-play.
Re: The Bitter Lesson
#6> Time spent on one is time not spent on the other. But from the AI researcher's view, "the other" doesn't require time; someone else is advancing the hardware, which is outside of the AI researcher's area. The general method to-be-run on better hardware is known today; it doesn't have to be researched. So should the AI researcher just twiddle their thumbs, waiting for the hardware to improve? In games like chess, it…
I think the author would agree with you, that improving chess-playing is not likely to be a productive domain for advancing AI in future, and for essentially the reasons you give here: it is too formal (this guess is based, in part, on some of the authors' other writing.)
Re: The Bitter Lesson
#7> Early methods conceived of vision as searching for edges, or generalized cylinders, or in terms of SIFT features. But today all this is discarded.
These aren't discarded, they are part of ML vision networks today. Edges are one of the 3x3 convolutions that a network can learn, SIFT/etc are the dense / clustering nets, I'll admit I just googled Generalized Cylinders (very interesting). There are others like SLAM as well.
Re: The Bitter Lesson
#8It's not just AI. Our trade's history is chockfull of valiant efforts to solve problems that were overrun by the exponential decline of computing costs by the time they really worked properly. Remember DSEE / ClearCase? They had all sorts of complicated virtual file systems to deliver tagged and branched contents of source code repositories. But drive space expanded with a Moore's-law style curve and now we have "git…
Ironically everything old is new again; now we have https://vfsforgit.org/ because keeping everything on one disk is too big, and OpenGL ES gets rid of immediate mode because communications with the host CPU is too slow.
Re: The Bitter Lesson
#9> They said that ``brute force" search may have won this time, but it was not a general strategy
It seems self-evident to me that 'brute force' is the most general strategy there is. Any (computable) problem is theoretically solvable by just coding the simplest, most obvious solution, which is usually pretty easy. The run-time of brute force is sometimes an issue, but that just means you need more of it!
Re: The Bitter Lesson
#10On the other hand at some point we will want AI to learn based on a small number of interactions. IE an AI that beats a human after playing 10 games of chess/starcraft etc. Right now it takes millions of training matches. Many real world situations don't happen that often so this fundamentally limits applications of the current generation of AI.