> After 15 minutes of confusion, it turned out Cloudflare had put a crazy robots.txt on my site without my consent (Cloudflare, love you guys, but this needs to stop). Might be the first time I see someone complain about their website being protected from a scraper, instead of the other way around.
I am pretty sure you have to enable cloudflare to manage your robots.txt, it shouldn't be doing that by itself. Maybe they did it by accident, it is just 1 click.
https://developers.cloudflare.com/changelog/post/2026-07-01-...
From Sept 15 all new sites added to CF will even block Googlebot by default on any page that serves ads as I understand it.
I think it's CF trying to force Google to separate out their bot traffic into bots for training and bots for the search index.
I think CF sees a big opportunity to get businesses to pay them to allow certain uses of their data but block others.
They're also starting a registry of "Approved" crawlers.