Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon's AI crawler is making my Git server unstable

xeiaso.net

Post reply on HN
Print view View original

11–20 of 261 posts

Re: Amazon's AI crawler is making my Git server unstable

#11

Post by xena »

AznHisoka wrote: post #9

Excuse my technical ignorance, but is it actually trying to get all the files in your git repo? Couldn’t you just have everything behind an user/pass if so?

Author of the article here. The behavior of the bot seems like this:

  while true {
    const page = await load_html_page(read_from_queue());
    save_somewhere(page);
    foreach link in page {
      enqueue(link);
    }
  }
This means that every link on every page gets enqueued and saved to do something. Naturally, this means that every file of every commit gets enqueued and scraped.

Having everything behind auth defeats the point of making the repos public.

Re: Amazon's AI crawler is making my Git server unstable

#12

Post by mschuster91 »

trebor wrote: post #3

Upvoted because we’re seeing the same behavior from all AI and Seo bots. They’re BARELY respecting Robots.txt, and hard to block. And when they crawl, they spam and drive up load so high they crash many servers for our clients. If AI crawlers want access they can either behave, or pay. The consequence will almost universal blocks otherwise!

Global tarpit is the solution. It makes sense anyway even without taking AI crawlers into account. Back when I had to implement that, I went the semi manual route - parse the access log and any IP address averaging more than X hits a second on /api gets a -j TARPIT with iptables [1].

Not sure how to implement it in the cloud though, never had the need for that there yet.

[1] https://gist.github.com/flaviovs/103a0dbf62c67ff371ff75fc62f...

Re: Amazon's AI crawler is making my Git server unstable

#13

Post by jsheard »

Ndymium wrote: post #8

I had this same issue recently. My Forgejo instance started to use 100 % of my home server's CPU as Claude and its AI friends from Meta and Google were hitting the basically infinite links at a high rate. I managed to curtail it with robots.txt and a user agent based blocklist in Caddy, but who knows how long that will work. Whatever happened to courtesy in scraping?

> Whatever happened to courtesy in scraping?

Money happened. AI companies are financially incentivized to take as much data as possible, as quickly as possible, from anywhere they can get it, and for now they have so much cash to burn that they don't really need to be efficient about it.

Re: Amazon's AI crawler is making my Git server unstable

#15

Post by keisborg »

thayne wrote: post #7
awsanswers wrote: post #4

Unacceptable, sorry this is happening. Do you know about fail2ban? You can have it automatically filter IPs that violate certain rules. One rule could be matching on the bot trying certain URLs. You might be able to get some kind of honeypot going with that idea. Good luck

They said that it is coming from different ip addresses every time, so fail2ban wouldn't help.

Monitor access logs for links that only crawlers can find.

Edit: oh, I got your point now.

Re: Amazon's AI crawler is making my Git server unstable

#16

Post by jsheard »

thayne wrote: post #7
awsanswers wrote: post #4

Unacceptable, sorry this is happening. Do you know about fail2ban? You can have it automatically filter IPs that violate certain rules. One rule could be matching on the bot trying certain URLs. You might be able to get some kind of honeypot going with that idea. Good luck

They said that it is coming from different ip addresses every time, so fail2ban wouldn't help.

Amazon does publish every IP address range used by AWS, so there is the nuclear option of blocking them all pre-emptively.

https://docs.aws.amazon.com/vpc/latest/userguide/aws-ip-rang...

Re: Amazon's AI crawler is making my Git server unstable

#17

Post by SteveNuts »

thayne wrote: post #7
awsanswers wrote: post #4

Unacceptable, sorry this is happening. Do you know about fail2ban? You can have it automatically filter IPs that violate certain rules. One rule could be matching on the bot trying certain URLs. You might be able to get some kind of honeypot going with that idea. Good luck

They said that it is coming from different ip addresses every time, so fail2ban wouldn't help.

It’ll most likely eventually help, as long as they don’t have an infinite address pool.

Do these bots use some client software (browser plugin, desktop app) that’s consuming unsuspecting users bandwidth for distributed crawling?

Re: Amazon's AI crawler is making my Git server unstable

#20

Post by xena »

jsheard wrote: post #16
thayne wrote: post #7

Earlier quoted context omitted.

They said that it is coming from different ip addresses every time, so fail2ban wouldn't help.

Amazon does publish every IP address range used by AWS, so there is the nuclear option of blocking them all pre-emptively. https://docs.aws.amazon.com/vpc/latest/userguide/aws-ip-rang...

I'd do that, but my DNS is via route 53. Blocking AWS would block my ability to manage DNS automatically as well as certificate issuance via DNS-01.
Post reply on HN