No evidence provided that this is amazonbot or AI related. Someone is just upset that their website is getting traffic, which seems asinine.
Amazon's AI crawler is making my Git server unstable
61–70 of 261 posts
Re: Amazon's AI crawler is making my Git server unstable
#62What is the proof that a hit from a residential IP address is actually Amazon? And if you have a way to tell, why not make use of it.
Re: Amazon's AI crawler is making my Git server unstable
#63Upvoted because we’re seeing the same behavior from all AI and Seo bots. They’re BARELY respecting Robots.txt, and hard to block. And when they crawl, they spam and drive up load so high they crash many servers for our clients. If AI crawlers want access they can either behave, or pay. The consequence will almost universal blocks otherwise!
Global tarpit is the solution. It makes sense anyway even without taking AI crawlers into account. Back when I had to implement that, I went the semi manual route - parse the access log and any IP address averaging more than X hits a second on /api gets a -j TARPIT with iptables [1]. Not sure how to implement it in the cloud though, never had the need for that there yet. [1] https://gist.github.com/flaviovs/103a0dbf6…
i would think public outcry by influencers on social media (such as this thread) is a better deterrent, and also establishes a public datapoint and exhibit for future reference.. as it is hard to scale the tarpit.
Re: Amazon's AI crawler is making my Git server unstable
#64[deleted]
Re: Amazon's AI crawler is making my Git server unstable
#65Upvoted because we’re seeing the same behavior from all AI and Seo bots. They’re BARELY respecting Robots.txt, and hard to block. And when they crawl, they spam and drive up load so high they crash many servers for our clients. If AI crawlers want access they can either behave, or pay. The consequence will almost universal blocks otherwise!
What do you mean by "barely" respecting robots.txt? Wouldn't that be more binary? Are they respecting some directives and ignoring others?
Amazonbot doesn't respect the `Crawl-Delay` directive. To be fair, Crawl-Delay is non-standard, but it is claimed to be respected by the other 3 most aggressive crawlers I see.
And how often does it check robots.txt? ClaudeBot will make hundreds of thousands of requests before it re-checks robots.txt to see that you asked it to please stop DDoSing you.
Re: Amazon's AI crawler is making my Git server unstable
#66What are the actual rules/laws about scraping? I have a few projects I'd like to do that involve scraping but have always been conscious about respecting the host's servers, plus whether private content is copyrighted. But sounds like AI companies don't give a shit lol. If anyone has a good resource on the subject I'd be grateful!
Re: Amazon's AI crawler is making my Git server unstable
#67[deleted]
Re: Amazon's AI crawler is making my Git server unstable
#68Re: Amazon's AI crawler is making my Git server unstable
#69Earlier quoted context omitted.
What do you mean by "barely" respecting robots.txt? Wouldn't that be more binary? Are they respecting some directives and ignoring others?
I believe that a number of AI bots only respect robot.txt entries that explicitly define their static user agent name. They ignore wildcards in user agents. That counts as barely imho. I found this out after OpenAI was decimating my site and ignoring the wildcard deny all. I had to add entires specifically for their three bots to get them to stop.
This is highly annoying and rude. Is there a complete list of all known bots and crawlers?
Re: Amazon's AI crawler is making my Git server unstable
#70I like the solution in this comment: https://news.ycombinator.com/item?id=42727510.
Put a link somewhere in your site that no human would visit, disallow it in robots.txt (under a wildcard because apparently OpenAI’s crawler specifically ignores wildcards), and when an IP address visits the link ban it for 24 hours.