Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon's AI crawler is making my Git server unstable

xeiaso.net

Post reply on HN
Print view View original

21–30 of 261 posts

Re: Amazon's AI crawler is making my Git server unstable

#21

Post by kgeist »

xena wrote: post #11
AznHisoka wrote: post #9

Excuse my technical ignorance, but is it actually trying to get all the files in your git repo? Couldn’t you just have everything behind an user/pass if so?

Author of the article here. The behavior of the bot seems like this: while true { const page = await load_html_page(read_from_queue()); save_somewhere(page); foreach link in page { enqueue(link); } } This means that every link on every page gets enqueued and saved to do something. Naturally, this means that every file of every commit gets enqueued and scraped. Having everything behind auth defeats the point of making…

>Having everything behind auth defeats the point of making the repos public.

Maybe add a captcha? Can be something simple and ad hoc, but unique enough to throw off most bots.

Re: Amazon's AI crawler is making my Git server unstable

#22

Post by gundmc »

trebor wrote: post #3

Upvoted because we’re seeing the same behavior from all AI and Seo bots. They’re BARELY respecting Robots.txt, and hard to block. And when they crawl, they spam and drive up load so high they crash many servers for our clients. If AI crawlers want access they can either behave, or pay. The consequence will almost universal blocks otherwise!

What do you mean by "barely" respecting robots.txt? Wouldn't that be more binary? Are they respecting some directives and ignoring others?

Re: Amazon's AI crawler is making my Git server unstable

#24

Post by unsnap_biceps »

xena wrote: post #20
jsheard wrote: post #16

Earlier quoted context omitted.

Amazon does publish every IP address range used by AWS, so there is the nuclear option of blocking them all pre-emptively. https://docs.aws.amazon.com/vpc/latest/userguide/aws-ip-rang...

I'd do that, but my DNS is via route 53. Blocking AWS would block my ability to manage DNS automatically as well as certificate issuance via DNS-01.

If you only block new inbound requests, it shouldn't impact your route 53 or DNS-01 usage.

Re: Amazon's AI crawler is making my Git server unstable

#25

Post by xena »

kgeist wrote: post #21
xena wrote: post #11

Earlier quoted context omitted.

Author of the article here. The behavior of the bot seems like this: while true { const page = await load_html_page(read_from_queue()); save_somewhere(page); foreach link in page { enqueue(link); } } This means that every link on every page gets enqueued and saved to do something. Naturally, this means that every file of every commit gets enqueued and scraped. Having everything behind auth defeats the point of making…

>Having everything behind auth defeats the point of making the repos public. Maybe add a captcha? Can be something simple and ad hoc, but unique enough to throw off most bots.

That's what I'm working on right now.

Re: Amazon's AI crawler is making my Git server unstable

#27

Post by Analemma_ »

Ndymium wrote: post #8

I had this same issue recently. My Forgejo instance started to use 100 % of my home server's CPU as Claude and its AI friends from Meta and Google were hitting the basically infinite links at a high rate. I managed to curtail it with robots.txt and a user agent based blocklist in Caddy, but who knows how long that will work. Whatever happened to courtesy in scraping?

The same thing that happened to courtesy in every other context: it only existed in contexts where there was no profit to be made in ignoring it. The instant that stopped being true, it was ejected.

Re: Amazon's AI crawler is making my Git server unstable

#28

Post by unsnap_biceps »

gundmc wrote: post #22
trebor wrote: post #3

Upvoted because we’re seeing the same behavior from all AI and Seo bots. They’re BARELY respecting Robots.txt, and hard to block. And when they crawl, they spam and drive up load so high they crash many servers for our clients. If AI crawlers want access they can either behave, or pay. The consequence will almost universal blocks otherwise!

What do you mean by "barely" respecting robots.txt? Wouldn't that be more binary? Are they respecting some directives and ignoring others?

I believe that a number of AI bots only respect robot.txt entries that explicitly define their static user agent name. They ignore wildcards in user agents.

That counts as barely imho.

I found this out after OpenAI was decimating my site and ignoring the wildcard deny all. I had to add entires specifically for their three bots to get them to stop.

Re: Amazon's AI crawler is making my Git server unstable

#29

Post by actuallyalys »

xena wrote: post #20
jsheard wrote: post #16

Earlier quoted context omitted.

Amazon does publish every IP address range used by AWS, so there is the nuclear option of blocking them all pre-emptively. https://docs.aws.amazon.com/vpc/latest/userguide/aws-ip-rang...

I'd do that, but my DNS is via route 53. Blocking AWS would block my ability to manage DNS automatically as well as certificate issuance via DNS-01.

They list a service for each address, so maybe you could block all the non-Route 53 IP addresses. Although that assumes they aren’t using the Route 53 IPs or unlisted IPs for scraping (the page warns it’s not a comprehensive list).

Regardless, it sucks that you have to deal with this. The fact that you’re a customer makes it all the more absurd.

Re: Amazon's AI crawler is making my Git server unstable

#30

Post by Vampiero »

trebor wrote: post #3

Upvoted because we’re seeing the same behavior from all AI and Seo bots. They’re BARELY respecting Robots.txt, and hard to block. And when they crawl, they spam and drive up load so high they crash many servers for our clients. If AI crawlers want access they can either behave, or pay. The consequence will almost universal blocks otherwise!

> The consequence will almost universal blocks otherwise!

Who cares? They've already scraped the content by then.

Post reply on HN