Do you reject them or put them in lower priority?
So far the AI bots are honest on the user agent. But my low budget server is significantly slower than it should be when I'm using it as a human.
Ask HN: AI companies' bots are making my server slow, what do you do?
1–6 of 6 posts
Re: Ask HN: AI companies' bots are making my server slow, what do you do?
#2good
Re: Ask HN: AI companies' bots are making my server slow, what do you do?
#3Re: Ask HN: AI companies' bots are making my server slow, what do you do?
#4Put it behind Cloudflare and try out the various firewall rules. Wrote this blog post covering some examples of rules and trade-offs https://developerwithacat.com/blog/202603/block-bots-scrapin...
Also post in r/Cloudflare subreddit. People are very helpful.
Re: Ask HN: AI companies' bots are making my server slow, what do you do?
#5https://anubis.techaro.lol/ or https://github.com/dgl/haphash/ .
[deleted]
Re: Ask HN: AI companies' bots are making my server slow, what do you do?
#6https://anubis.techaro.lol/ or https://github.com/dgl/haphash/ .
I saw them used in the linux community, maybe it's finally my time to use/need this kind of tools.
(At the same time, I'm happy if my site can help the AIs / its users, but I prefer it not significantly slow down my own usage. There should be a balanced combination)