Perhaps the Bitter Lesson, more data and compute triumph over intelligent algorithms, is true only in a certain range, and a new intelligent algorithm is needed to go beyond. For example, nearest neighbors algorithm obey the Bitter Lesson depending of the relation between grid of data points and complexity of the problem.
“The Bitter Lesson” is wrong. Well sort of
21–30 of 35 posts
Re: “The Bitter Lesson” is wrong. Well sort of
#22The Bitter Lesson is saying, if you're going to use human knowledge, be aware that your solution is temporary. It's not wrong. And it's not wrong to use human knowledge to solve your "today" problem.
Re: “The Bitter Lesson” is wrong. Well sort of
#23it should be noted that "The Bitter Lesson" is not a general principle, for example LLMs are not able to sum 1000 digits numbers, but a python program with a few lines can do it. Also we provide tools (algorithms) for LLMs, these examples show that "The Bitter Lesson" is not true beyond a certain context.
Re: “The Bitter Lesson” is wrong. Well sort of
#24From TFA:
> No machine learning model was ever built using pure “human knowledge” — because then it wouldn’t be a learning model. It would be a hard coded algorithm.
I guess the author hasn't heard of expert systems? Systems like MYCIN (https://en.wikipedia.org/wiki/Mycin) were heralded as incredible leaps forward at the time, and they indeed consisted of pure “human knowledge.”
I am disturbed whenever a thinkpiece is written by someone who obviously didn't do their research.
Re: “The Bitter Lesson” is wrong. Well sort of
#25From TFA: > No machine learning model was ever built using pure “human knowledge” — because then it wouldn’t be a learning model. It would be a hard coded algorithm. I guess the author hasn't heard of expert systems? Systems like MYCIN ( https://en.wikipedia.org/wiki/Mycin ) were heralded as incredible leaps forward at the time, and they indeed consisted of pure “human knowledge.” I am disturbed whenever a thinkpiece…
Expert systems aren't a machine learning approach even if they're an AI approach.
Re: “The Bitter Lesson” is wrong. Well sort of
#26The Leela Chess Zero vs Stockfish case also offers an interesting perspective on the bitter lesson. Here's my (maybe a bit loose) recollection of what happened: Step 1- Stockfish was the typical human-knowledge AI, with tons of actual chess knowledge injected in the process of building an efficient chess engine. Step 2. Then came Leela Chess Zero, with its Alpha Zero-inspired training, a chess engine trained fully wi…
Stockfish has got rid of old handwritten evaluation now.
https://github.com/official-stockfish/Stockfish/pull/4674
Its evaluation now purely relies on NNUE neural network.
So it's an good exmaple of the better lesson. More compute evently won against handwritten evaluation. Stockfish developers thought old evaluation would help neural network so they kept the code for a few years, then it turned out that NNUE neural network didn't need any input of human chess knowledge.
Re: “The Bitter Lesson” is wrong. Well sort of
#27From TFA: > No machine learning model was ever built using pure “human knowledge” — because then it wouldn’t be a learning model. It would be a hard coded algorithm. I guess the author hasn't heard of expert systems? Systems like MYCIN ( https://en.wikipedia.org/wiki/Mycin ) were heralded as incredible leaps forward at the time, and they indeed consisted of pure “human knowledge.” I am disturbed whenever a thinkpiece…
Expert systems aren't a machine learning approach even if they're an AI approach.
The original article is about AI. Then about machine learning. Then about AI...
The author doesn't seem to make up his mind about it. Or the article is AI-generated slop maybe.
Re: “The Bitter Lesson” is wrong. Well sort of
#28From TFA: > No machine learning model was ever built using pure “human knowledge” — because then it wouldn’t be a learning model. It would be a hard coded algorithm. I guess the author hasn't heard of expert systems? Systems like MYCIN ( https://en.wikipedia.org/wiki/Mycin ) were heralded as incredible leaps forward at the time, and they indeed consisted of pure “human knowledge.” I am disturbed whenever a thinkpiece…
Expert systems aren't a machine learning approach even if they're an AI approach.
Yes, but the Bitter Lesson is about AI, not ML.
Re: “The Bitter Lesson” is wrong. Well sort of
#29I first read the "Blitter" lesson and got excited for a moment.
Re: “The Bitter Lesson” is wrong. Well sort of
#30Earlier quoted context omitted.
> you're building intelligence without understanding why it works. But if we do a good enough job of that, it should then be able to explain to us why it works (after it does some research/science on itself). Yes?
Bit fantastical. We are a general intelligence and we dont understand ourselves
Indeed. But the premise of the objection, was that it is understandable, and a shame that we're not putting such understanding before implementing these systems.
If you're right, and it's essentially impossible to understand (and we still want to advance these technologies) we will have to do so in some degree of ignorance anyway.