It's so cute how people think anyone obeys robots.txt :)
And why should anyone believe you when you didn’t give any evidence of anything?
Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt?
31–33 of 33 posts
Re: Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt?
#32Earlier quoted context omitted.
It's a free wiki so not visiting is a right you have, sure .. but also OP shouldn't have to spend loads of $ to host a free wiki and at the same time support a very bespoke thing such as non standard user agents.
> is a right I believe they're saying that cloudflare will block them just for using a blacklisted client, even if they're legit users and not bots
Not even that; it's a whitelist, not a blacklist, and the only clients whitelisted are essentially those of Big Browser.