Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon's AI crawler is making my Git server unstable

xeiaso.net

Post reply on HN
Print view View original

231–240 of 261 posts

Re: Amazon's AI crawler is making my Git server unstable

#231

Post by jsheard »

andrethegiant wrote: post #177

I’m working on a centralized platform[1] to help web crawlers be polite by default by respecting robots.txt, 429s, etc, and sharing a platform-wide TTL cache just for crawlers. The goal is to reduce global bot traffic by providing a convenient option to crawler authors that makes their bots play nice with the open web. [1] https://crawlspace.dev

> respecting robots.txt

What agent name should we put in robots.txt to deny your crawler without using a wildcard? I can't see that documented anywhere.

Re: Amazon's AI crawler is making my Git server unstable

#232

Post by s1mplicissimus »

dbaio wrote: post #58

suffering with it as well. why can't they just `git clone` and do their stuff? =)

My guess is: Because the cloning is done by a stochastic parrot and the people running the parrot don't have the slightest idea what they are doing

Re: Amazon's AI crawler is making my Git server unstable

#233

Post by genter »

Reason077 wrote: post #217
Animats wrote: post #132

It's time for a lawyer letter. See the Computer Fraud and Abuse Act prosecution guidelines.[1] In general, the US Justice Department will not consider any access to open servers that's not clearly an attack to be "unauthorized access". But, "However, when authorizers later expressly revoke authorization—for example, through unambiguous written cease and desist communications that defendants receive and understand—the…

Do we need a "robots must respect robots.txt" law?

Corporations expect average people to read and abide by a ten thousand line EULA, yet it's too much work for them to respect a trivially parseable text file.

Re: Amazon's AI crawler is making my Git server unstable

#235

Post by jdjdjdjdhy »

qwaxys wrote: post #223

Nothing wrong with putting a link to a zip bomb in your robots.txt and disallowing them. Also, just put a limit on requests per IP? https://nginx.org/en/docs/http/ngx_http_limit_req_module.htm...

How would that help you? Bots generally don't care about zip contents

Re: Amazon's AI crawler is making my Git server unstable

#236

Post by jdjdjdjdhy »

evantbyrne wrote: post #100

It seems like git self-hosters frequently encounter DDoS issues. I know it's not typical for free software, but I wonder if gating file contents behind a login and allowing registrations could be the answer for self-hosting repositories on the cheap.

Self hosting a public git is as cheap as it can get, since it's completely static content

Sure you might get bandwidth saturated, but that can happen with any type of content

Re: Amazon's AI crawler is making my Git server unstable

#237

Post by marginalia_nu »

vachina wrote: post #108

I’m surprised everyone else’s servers are struggling to handle a couple of bot scrapes. I run a couple of public facing websites on a NUC and it just… chugs along? This is also amidst the constant barrage of OSINT attempts at my IP.

Crawling a git repository can be pretty resource intensive on the server side. Viewing the state of HEAD is pretty fast, but once you go deep into historical commits, it requires considerable processing to reconstruct the historical state of the repository.

Re: Amazon's AI crawler is making my Git server unstable

#238

Post by smarnach »

Animats wrote: post #139
gundmc wrote: post #22

Earlier quoted context omitted.

What do you mean by "barely" respecting robots.txt? Wouldn't that be more binary? Are they respecting some directives and ignoring others?

Here's Google, complaining of problems with pages they want to index but I blocked with robots.txt. New reason preventing your pages from being indexed Search Console has identified that some pages on your site are not being indexed due to the following new reason: Indexed, though blocked by robots.txt If this reason is not intentional, we recommend that you fix it in order to get affected pages indexed and appearing…

They are not complaining. You configured Google Search Console to notify you about problems that affect the search ranking of your site, and that's what they do. if you don't want to receive these messages, turn them off in Google Search Console.

Re: Amazon's AI crawler is making my Git server unstable

#239

Post by evantbyrne »

jdjdjdjdhy wrote: post #236
evantbyrne wrote: post #100

It seems like git self-hosters frequently encounter DDoS issues. I know it's not typical for free software, but I wonder if gating file contents behind a login and allowing registrations could be the answer for self-hosting repositories on the cheap.

Self hosting a public git is as cheap as it can get, since it's completely static content Sure you might get bandwidth saturated, but that can happen with any type of content

It's a lot less bandwidth to serve one login screen than the entire contents of all of your source repositories to a bunch of crawlers. You can also cache that one login screen in memory.

Re: Amazon's AI crawler is making my Git server unstable

#240

Post by YetAnotherNick »

sigmoid10 wrote: post #227
Aurornis wrote: post #159

Earlier quoted context omitted.

> Then you wait and see if Amazon stops. That’s if the requests are actually coming from Amazon, which seems very unlikely given some of the details in the post (rotating user agents, residential IPs, seemingly not interpreting robots.txt). The Amazon bot should come from known Amazon IP ranges and respect robots.txt. An Amazon engineer confirmed it in another comment: https://news.ycombinator.com/item?id=42751729 Th…

I wonder how the author hasn't reached this conclusion. The official Amazon Crawler docs literally tell you how to distinguish between legit Amazonbots and malicious copycats via DNS lookup: https://developer.amazon.com/amazonbot

Why would someone copycat amazonbot?
Post reply on HN