Amazon's AI crawler is making my Git server unstable
1–10 of 261 posts
Re: Amazon's AI crawler is making my Git server unstable
#2Re: Amazon's AI crawler is making my Git server unstable
#3Upvoted because we’re seeing the same behavior from all AI and Seo bots. They’re BARELY respecting Robots.txt, and hard to block. And when they crawl, they spam and drive up load so high they crash many servers for our clients.
If AI crawlers want access they can either behave, or pay. The consequence will almost universal blocks otherwise!
Re: Amazon's AI crawler is making my Git server unstable
#4Unacceptable, sorry this is happening. Do you know about fail2ban? You can have it automatically filter IPs that violate certain rules. One rule could be matching on the bot trying certain URLs. You might be able to get some kind of honeypot going with that idea. Good luck
Re: Amazon's AI crawler is making my Git server unstable
#5Has anyone tried using Cloudflare Bot Management and how effective is it for such bots ?
Re: Amazon's AI crawler is making my Git server unstable
#6> About 10% of the requests do not have the amazonbot user agent.
Is there any bot string in the user agent? I'd wonder if it's GPTBot as I believe they don't respect a robots.txt deny wildcard.
Re: Amazon's AI crawler is making my Git server unstable
#7Unacceptable, sorry this is happening. Do you know about fail2ban? You can have it automatically filter IPs that violate certain rules. One rule could be matching on the bot trying certain URLs. You might be able to get some kind of honeypot going with that idea. Good luck
They said that it is coming from different ip addresses every time, so fail2ban wouldn't help.
Re: Amazon's AI crawler is making my Git server unstable
#8I had this same issue recently. My Forgejo instance started to use 100 % of my home server's CPU as Claude and its AI friends from Meta and Google were hitting the basically infinite links at a high rate. I managed to curtail it with robots.txt and a user agent based blocklist in Caddy, but who knows how long that will work.
Whatever happened to courtesy in scraping?
Re: Amazon's AI crawler is making my Git server unstable
#9Excuse my technical ignorance, but is it actually trying to get all the files in your git repo? Couldn’t you just have everything behind an user/pass if so?
Re: Amazon's AI crawler is making my Git server unstable
#10> About 10% of the requests do not have the amazonbot user agent. Is there any bot string in the user agent? I'd wonder if it's GPTBot as I believe they don't respect a robots.txt deny wildcard.
[deleted]