Using a different definition of Google than TFA : We have discussed recently a huge issue preventing new Google-like web crawling competitors from emerging : If you're not GAFAM(-like), and so aren't whitelisted, then anti-spam/DDoS filters like Cloudflare will simply prevent your crawler from working !
I've been working on a project that involves executing web crawling on a responsible scale but this has concerned me when I ramp to wider/regular fetching how will it be received by the web distribution mafia. I recall last time i set up my own mail server that it was very hard to do it in such a manner that emails from it were not automatically listed as spam
One of the tragedies of the tech powerhouses is that their decision making that effects the rest of us is often thinly-veiled power tripping that they've convinced themselves is for the good of everyone. When in reality, they just haven't delivered good enough software to eliminate the problem entirely.
It's like a large scale version of a bad corporate computer security team. They can't quite design all the inconveniences out of their "solutions." Which is normally fine (after all what software and/or human-managed process is perfect the first time?), but out of a refusal to admit to their limitations they've convinced themselves that bureaucratic interactions with them to get day-to-day work done are necessary and important features of the design.
There are genuinely technical challenges with no good technical solution, so political ones are employed. But sometimes you start getting too many middle managers that think the answer to every problem that requires systems thinking is just more self-assured tough-talking or "we could make more money by NOT fixing this." And then suddenly all the nice properties of the system slowly start disappearing or have conditional access gates.