Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 33 posts

Re: Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt?

#25

Post by foreigner »

keisborg wrote: post #19

An old trick is to add a page to the robots disallow list, but the page should also be findable by crawlers. If a bot visits this page, you know it’s a bad actor.

I love this idea! Can anybody else with experience with this chime in? Does it actually work?

Re: Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt?

#26

Post by trod1234 »

Most of the major players seem to be actively ignoring robots.txt.

They claim they don't do this, but those claims amount to lies. The access logs shows what they actually do, and its gotten so egregious that some sites have buckled under resource exhaustion caused by these bad actors. (DDOS attack).

Re: Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt?

#27

Post by blakesterz »

I found that robots.txt does an OK job. It doesn't block everything, but I wouldn't run without one because many really busy bots do follow the rules. It's simple, cheap and does knock out a bunch of traffic. The AWS WAF has some challenge rules that I found work great stopping some of the other crappy and aggressive bots. And then the last line of defense is some addition nginx blocks. Those three layers really got things under control.

Re: Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt?

#28

Post by zaphodias »

aetherspawn wrote: post #18
userbinator wrote: post #15

Earlier quoted context omitted.

More precisely described as "block non-sanctioned user agents". Using that feature will ensure I never visit your site again.

It's a free wiki so not visiting is a right you have, sure .. but also OP shouldn't have to spend loads of $ to host a free wiki and at the same time support a very bespoke thing such as non standard user agents.

> is a right

I believe they're saying that cloudflare will block them just for using a blacklisted client, even if they're legit users and not bots

Re: Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt?

#29

Post by taftster »

scoofy wrote: post #20
taftster wrote: post #17

Earlier quoted context omitted.

Right?! robots.txt is basically a map to the inner secret parts of your website, the "real" content. Robots.txt is basically treated as "these rules are for thee, not for me" access control. If you don't want your content crawled, you really need to put it behind a login of some sort (and watch it still get crawled regardless) or just not publish it at all. Sad state we're in.

I'm getting close to a building a honey pot, because I want responsible robots to be able to crawl my site! It's about sharing information after all. If it gets large enough to have a positive monetary benefit to me, I'd happily let the ai bots crawl it responsibly too. It's a wiki, so I don't want it to me my information, ever. I'm just the one footing the bill right now.

Yeah, I used to do that. I'd put bad actors into a randomly generated never ending link farm. I'd put a special link in my robots.txt and any bot that visited it, I would assume was a bad actor. This link was not accessible from any other location.

The results were striking, how many bots just stupidly tumbled down the random link rabbit hole. I'm not sure what they were searching for, but all they got from me was gobbledygook text and more links.

Post reply on HN