> After 15 minutes of confusion, it turned out Cloudflare had put a crazy robots.txt on my site without my consent (Cloudflare, love you guys, but this needs to stop). Might be the first time I see someone complain about their website being protected from a scraper, instead of the other way around.
I think the issue is the lack of consent. Whether a service I use is protecting my website from scrapers or feeding everything to scrapers, some of us would prefer that it takes our informed consent before doing so.
FWIW, I just set up a domain last week, and the web UI asked if I want to block AI crawlers or not.
Perhaps OP set it up agentically, and the agent didn’t pass an optional param correctly, or ticked the box for him?