My theory is that Google lost the spam war. All the leetcode in the world won't teach someone to build an ML model that can tell whether a website is spam. So Google outsourced the spam detection problem -- they heavily bias their results towards only the most popular sites, who either have human moderators or paid contributors, and those sites do the spam management that Google's automated approach is incapable of.
It is more than the spam war...a lot of content is just no longer produced and exposed to the open web (think about how much content goes into tiktok, discord, etc and you will never get that into your search results). Google has less useful content to index, algorithms can't fix that. There is more spam only because that's the only open content that gets added massively. The winners of this battle will be the places…
I see this all the time when trying to find information about old computers. So many of the good vintage computing resources are locked in social services or mailing lists that the information never shows up in search engines.
It feels a lot like the days when information was balkanized between AOL, GEnie, CompuServe, American PeopleLink, Delphi, etc.
Search engines were supposed to fix that and make all the world's information discoverable. They didn't.