Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt? • Hacker News Board

Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt?

#1

Post by scoofy » Wed, Jan 29, 2025, 3:09 AM UTC

My niche little site, http://golfcourse.wiki seems to be very popular with AI bots. They basically become most of my traffic. Most of them follow robots.txt, and that's nice and all, but they are costing me non-trivial amounts of money.

I don't want to block most search engines. I don't want to block legitimate institutions like archive.org. Is there a whitelist that I could crib instead of pretty much having to update my robots file every damn day?

Re: Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt?

#2

Post by anxixddjs » Wed, Jan 29, 2025, 3:20 AM UTC

sorry cant help - but really cool site!

Re: Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt?

#3

Post by andrewfromx » Wed, Jan 29, 2025, 3:39 AM UTC

robots.txt will just be ignored by bad actors.

And the user-agent sent is completely in control of the bad actor. They can send their user-agent as "google bot".

You would need something like WAF from https://www.cloudflare.com/ or AWS

Re: Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt?

#4

Post by palmfacehn » Wed, Jan 29, 2025, 4:11 AM UTC

Blocking Amazon, Huawei, China Telecom and few other ASNs did it for me. You should be able to cut, sort -u your log files to find the biggest offenders by user-agent if they truly obey robots.txt.

Re: Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt?

#5

Post by captn3m0 » Wed, Jan 29, 2025, 4:15 AM UTC

I’ve been using https://github.com/ai-robots-txt/ai.robots.txt on my Gitea instance.

Re: Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt?

#6

Post by _blk » Wed, Jan 29, 2025, 4:16 AM UTC

Takes balls to put the URL in a HN post if you're looking to reduce traffic costs.. Of course it's legit traffic for once but I assume curiosity just got the better half of them.. .. On the constructive side: shield it with cloudflare.

Re: Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt?

#7

Post by aetherspawn » Wed, Jan 29, 2025, 4:26 AM UTC

Use Cloudflare (free) and just disable bots or add a captcha.

You'll probably also save money if you enable and configure free caching.

Re: Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt?

#8

Post by scoofy » Wed, Jan 29, 2025, 4:26 AM UTC

_blk wrote: post #6

Takes balls to put the URL in a HN post if you're looking to reduce traffic costs.. Of course it's legit traffic for once but I assume curiosity just got the better half of them.. .. On the constructive side: shield it with cloudflare.

I’m fine with a burst of traffic… that’s what it’s designed for. What it’s not designed for is 24-7-365 being slowly pinged on every single page (thousands), by every single robot (thousands).

Re: Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt?

#9

Post by scoofy » Wed, Jan 29, 2025, 4:28 AM UTC

aetherspawn wrote: post #7

Use Cloudflare (free) and just disable bots or add a captcha. You'll probably also save money if you enable and configure free caching.

Will this still allow Google and Bing and Archive bots?

Re: Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt?

#10

Post by aetherspawn » Wed, Jan 29, 2025, 4:48 AM UTC

scoofy wrote: post #9

aetherspawn wrote: post #7

Use Cloudflare (free) and just disable bots or add a captcha. You'll probably also save money if you enable and configure free caching.

Will this still allow Google and Bing and Archive bots?

I quickly checked for you, and my Cloudflare account currently has the following option: "Block AI Bots" under xxx.com > Security > Bots which I think does what you want.

I am not paying for Cloudflare and it allows me to enable this option. If you use the (other) Bot Fight Mode or captcha options, then yes, you will block crawlers. AFAIK specific bot user agents can also be blocked (or allowed) using a Page Rule, which is also a free tier feature.