Druids used to decry that literacy caused people to lose their ability to memorize sacred teachings. And they’re right! But literacy still happened and we’re all either dumber or smarter for it.
It's more complex than that. The three pillars of learning are theory (finding out about the thing), practice (doing the thing) and metacognition (being right, or more importantly, wrong. And correcting yourself.). Each of those steps reinforce neural pathways. They're all essential in some form or another. Literacy, books, saving your knowledge somewhere else removes the burden of remembering everything in your head…
I feel as though this analysis only makes sense in hindsight; in the past, when people used to say using books as a way to store knowledge outside your brain would make a similar argument, but would add a fourth pillar of memorization. Even now, a lot of people in different professions (such as law, and medicine) still absolutely drill memorization as the first step in building a strong knowledge base before getting into more practical, day-to-day used information. When people are forced to memorize a large amount of things in a cohesive subject, it forces your brain to make connections between ideas out of necessity to keep the information in your head. This definitely has an effect on metacognition and practice. So I wouldn't argree with you that the analogy with books=brain rot isn't valid.