We have discussed recently a huge issue preventing new Google-like web crawling competitors from emerging :
If you're not GAFAM(-like), and so aren't whitelisted, then anti-spam/DDoS filters like Cloudflare will simply prevent your crawler from working !