Robots.txt is a joke, use Cloudflare's Block AI Bots feature if you are using Cloudflare.
Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt?
11–20 of 33 posts
Re: Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt?
#12I operate under the assumption that Google, OpenAI, Anthropic, Bytedance et al either totally ignore it or only follow it selectively. I haven't touched my Robots.txt in a while. Instead, I have nginx return empty and/or bogus responses when it sees those UA substrings.
Re: Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt?
#13Robots.txt is a joke, use Cloudflare's Block AI Bots feature if you are using Cloudflare.
On this point, if you turn on bot fight mode it also says it blocks verified bots.
But, bot fight mode says "there is a newer version of this setting" however it does not link to it.
Anyone have any insight on the blocked verified bots or the supposed new version?
Re: Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt?
#14Just ratelimit.
Re: Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt?
#15Robots.txt is a joke, use Cloudflare's Block AI Bots feature if you are using Cloudflare.
More precisely described as "block non-sanctioned user agents".
Using that feature will ensure I never visit your site again.
Re: Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt?
#16Takes balls to put the URL in a HN post if you're looking to reduce traffic costs.. Of course it's legit traffic for once but I assume curiosity just got the better half of them.. .. On the constructive side: shield it with cloudflare.
I’m fine with a burst of traffic… that’s what it’s designed for. What it’s not designed for is 24-7-365 being slowly pinged on every single page (thousands), by every single robot (thousands).
Maybe the bots are hallucinating off-by-one errors and then 25-8-367 isn’t all that bad.
Re: Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt?
#17I operate under the assumption that Google, OpenAI, Anthropic, Bytedance et al either totally ignore it or only follow it selectively. I haven't touched my Robots.txt in a while. Instead, I have nginx return empty and/or bogus responses when it sees those UA substrings.
Right?! robots.txt is basically a map to the inner secret parts of your website, the "real" content. Robots.txt is basically treated as "these rules are for thee, not for me" access control.
If you don't want your content crawled, you really need to put it behind a login of some sort (and watch it still get crawled regardless) or just not publish it at all. Sad state we're in.
Re: Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt?
#18Robots.txt is a joke, use Cloudflare's Block AI Bots feature if you are using Cloudflare.
More precisely described as "block non-sanctioned user agents". Using that feature will ensure I never visit your site again.
It's a free wiki so not visiting is a right you have, sure .. but also OP shouldn't have to spend loads of $ to host a free wiki and at the same time support a very bespoke thing such as non standard user agents.
Re: Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt?
#19An old trick is to add a page to the robots disallow list, but the page should also be findable by crawlers. If a bot visits this page, you know it’s a bad actor.
Re: Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt?
#20I operate under the assumption that Google, OpenAI, Anthropic, Bytedance et al either totally ignore it or only follow it selectively. I haven't touched my Robots.txt in a while. Instead, I have nginx return empty and/or bogus responses when it sees those UA substrings.
Right?! robots.txt is basically a map to the inner secret parts of your website, the "real" content. Robots.txt is basically treated as "these rules are for thee, not for me" access control. If you don't want your content crawled, you really need to put it behind a login of some sort (and watch it still get crawled regardless) or just not publish it at all. Sad state we're in.
I'm getting close to a building a honey pot, because I want responsible robots to be able to crawl my site! It's about sharing information after all. If it gets large enough to have a positive monetary benefit to me, I'd happily let the ai bots crawl it responsibly too.
It's a wiki, so I don't want it to me my information, ever. I'm just the one footing the bill right now.