21–30 of 35 posts
Related: The Bitter Lesson (2019) - https://news.ycombinator.com/item?id=30889873 - April 2022 (37 comments) The Bitter Lesson - https://news.ycombinator.com/item?id=28409314 - Sept 2021 (1 comment) The Bitter Lesson (From AI Research) - https://news.ycombinator.com/item?id=27924335 - July 2021 (1 comment) The Bitter Lesson (2019) - https://news.ycombinator.com/item?id=23781400 - July 2020 (85 comments) The Bitter Le…
I think this bitter lesson needs to be taken for a several grains of salt. Number one, the progress in a particular AI field tends to go, at first, from custom to more general algorithms, exactly as Professor Richard Sutton described. However, there is a second part to this progress, where, once we "understood" (which we never really do) the new level of general algorithms (say Transformers in NLP), we begin to put b…
I don’t follow your critique for #2. SVMs, Random forests, etc., aren’t the counterexample to Rich’s post (for anyone who knows him, Rich doesn’t even particularly _like_ neural networks). The counterexample is hand crafted features. A counter example would be showing a number of successful examples in, say, computer vision, where handcrafted features do better than learned features. This is largely not the case. In,…
Earlier quoted context omitted.
I don’t follow your critique for #2. SVMs, Random forests, etc., aren’t the counterexample to Rich’s post (for anyone who knows him, Rich doesn’t even particularly _like_ neural networks). The counterexample is hand crafted features. A counter example would be showing a number of successful examples in, say, computer vision, where handcrafted features do better than learned features. This is largely not the case. In,…
Thank you for the good point! I edited the comment.
I think this bitter lesson needs to be taken for a several grains of salt. Number one, the progress in a particular AI field tends to go, at first, from custom to more general algorithms, exactly as Professor Richard Sutton described. However, there is a second part to this progress, where, once we "understood" (which we never really do) the new level of general algorithms (say Transformers in NLP), we begin to put b…
> [the critics of brute force] said that ``brute force" search may have won this time, but it was not a general strategy, and anyway it was not how people played chess. No expert in AI or chess, but I assume that this observation is wrong. Chess players do indeed construct moves based on a search of an interior mental library... a sort of pattern recognition. Any chess players here agree/disagree? As an artist who is…
The AI art on the internet is (in most cases) cherry-picked, and therefore reasonable fodder for further training.
Turns out it might be even easier to build a brain than to understand one.
Put it this way. Do you believe premature optimization is the root of all evil? Then why do you believe that some subtle, intricate optimization for machine intelligence will win out over brute compute force? It's not that there's optimizations to be done, it's knowing what optimizations will yield the most value after the space is explored with better computing capabilities.
To me, this is almost like a generalized "Proebstings Law" [0], where compiler optimizations give a doubling every 18 years compared, to say, some type of generalized Moore's law which give roughly a doubling in compute every 1.5 to 2 years.