Earlier quoted context omitted.
I believe that a number of AI bots only respect robot.txt entries that explicitly define their static user agent name. They ignore wildcards in user agents. That counts as barely imho. I found this out after OpenAI was decimating my site and ignoring the wildcard deny all. I had to add entires specifically for their three bots to get them to stop.
This is highly annoying and rude. Is there a complete list of all known bots and crawlers?
Amazon's AI crawler is making my Git server unstable
71–80 of 261 posts
Re: Amazon's AI crawler is making my Git server unstable
#72Can demonstrable ignoring of robots.txt help the cases of copyright infringement lawsuits against the "AI" companies, their partners, and customers?
On what legal basis?
In the UK, the Computer Misuse Act applies if:
* There is knowledge that the intended access was unauthorised
* There is an intention to secure access to any program or data held in a computer
I imagine US law has similar definitions of unauthorized access?
`robots.txt` is the universal standard for defining what is unauthorised access for bots. No programmer could argue they aren't aware of this, and ignoring it, for me personally, is enough to show knowledge that the intended access was unauthorised. Is that enough for a court? Not a goddamn clue. Maybe we need to find out.
Re: Amazon's AI crawler is making my Git server unstable
#73Earlier quoted context omitted.
Global tarpit is the solution. It makes sense anyway even without taking AI crawlers into account. Back when I had to implement that, I went the semi manual route - parse the access log and any IP address averaging more than X hits a second on /api gets a -j TARPIT with iptables [1]. Not sure how to implement it in the cloud though, never had the need for that there yet. [1] https://gist.github.com/flaviovs/103a0dbf6…
One such tarpit (Nepenthes) was just recently mentioned on Hacker News: https://news.ycombinator.com/item?id=42725147 Their site is down at the moment, but luckily they haven't stopped Wayback Machine from crawling it: https://web.archive.org/web/20250117030633/https://zadzmo.or...
Quixotic[0] (my content obfuscator) includes a tarpit component, but for something like this, I think the main quixotic tool would be better - you run it against your content once, and it generates a pre-obfuscated version of it. It takes a lot less of your resources to serve than dynamically generating the tarpit links and content.
Re: Amazon's AI crawler is making my Git server unstable
#74Earlier quoted context omitted.
On what legal basis?
In the UK, the Computer Misuse Act applies if: * There is knowledge that the intended access was unauthorised * There is an intention to secure access to any program or data held in a computer I imagine US law has similar definitions of unauthorized access? `robots.txt` is the universal standard for defining what is unauthorised access for bots. No programmer could argue they aren't aware of this, and ignoring it, fo…
> `robots.txt` is the universal standard
Quite the assumption, you just upset a bunch of alien species.
Re: Amazon's AI crawler is making my Git server unstable
#75I had this same issue recently. My Forgejo instance started to use 100 % of my home server's CPU as Claude and its AI friends from Meta and Google were hitting the basically infinite links at a high rate. I managed to curtail it with robots.txt and a user agent based blocklist in Caddy, but who knows how long that will work. Whatever happened to courtesy in scraping?
Mind sharing a decent robots.txt and/or user-agent list to block the AI crawlers?
Re: Amazon's AI crawler is making my Git server unstable
#76Has anyone tried using Cloudflare Bot Management and how effective is it for such bots ?
I put my personal site behind Cloudflare last year specifically to combat AI bots. It's very effective, but I hate that the web has devolved to a state where using a service like Cloudflare is practically no longer optional.
Re: Amazon's AI crawler is making my Git server unstable
#77Earlier quoted context omitted.
You should check your websites like grass dot io (I refuse to give them traffic). They pay you for your bandwidth while they resell it to 3rd parties, which is why a lot of bot traffic looks like it comes from residential IPs.
Yes, but the point is that big company crawlers aren’t paying for questionably sourced residential proxies. If this person is seeing a lot of traffic from residential IPs then I would be shocked if it’s really Amazon. I think someone else is doing something sketchy and they put “AmazonBot” in the user agent to make victims think it’s Amazon. You can set the user agent string to anything you want, as we all know.
It’s not residential proxies. It’s Amazon using IPs they sublease from residential ISPs.
Re: Amazon's AI crawler is making my Git server unstable
#78[flagged]
Personally I'm not trying to block the bots, I'm trying to avoid the bandwidth bill.
I've recently blocked everything that isn't offering a user agent. If it had only pulled text I probably wouldn't have cared, but it was pulling images as well (bot designers, take note - you can have orders of magnitude less impact if you skip the images).
For me personally, what's left isn't eating enough bandwidth for me to care, and I think any attempt to serve some bots is doomed to failure.
If I really, really hated chatbots (I don't), I'd look at approaches that poison the well.
Re: Amazon's AI crawler is making my Git server unstable
#79Earlier quoted context omitted.
You should check your websites like grass dot io (I refuse to give them traffic). They pay you for your bandwidth while they resell it to 3rd parties, which is why a lot of bot traffic looks like it comes from residential IPs.
Yes, but the point is that big company crawlers aren’t paying for questionably sourced residential proxies. If this person is seeing a lot of traffic from residential IPs then I would be shocked if it’s really Amazon. I think someone else is doing something sketchy and they put “AmazonBot” in the user agent to make victims think it’s Amazon. You can set the user agent string to anything you want, as we all know.
I wonder if anyone has checked whether Alexa devices serve as a private proxy network for AmazonBot’s use.
Re: Amazon's AI crawler is making my Git server unstable
#80Can demonstrable ignoring of robots.txt help the cases of copyright infringement lawsuits against the "AI" companies, their partners, and customers?
Probably not copyright infringement. But it is probably (hopefully?) a violation of CFAA, both because it is effectively DDoSing you, and they are ignoring robots.txt.
Maybe worth contacting law enforcement?
Although it might not actually be Amazon.