Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon's AI crawler is making my Git server unstable

xeiaso.net

Post reply on HN
Print view View original

221–230 of 261 posts

Re: Amazon's AI crawler is making my Git server unstable

#221

Post by WhyNotHugo »

Reason077 wrote: post #217
Animats wrote: post #132

It's time for a lawyer letter. See the Computer Fraud and Abuse Act prosecution guidelines.[1] In general, the US Justice Department will not consider any access to open servers that's not clearly an attack to be "unauthorized access". But, "However, when authorizers later expressly revoke authorization—for example, through unambiguous written cease and desist communications that defendants receive and understand—the…

Do we need a "robots must respect robots.txt" law?

If we did, bot authors would comply by just changing their User-Agent to something different that’s not expressly forbidden.

(Disallowing * isn’t usually an option since it makes you disappear from search engines).

Re: Amazon's AI crawler is making my Git server unstable

#222

Post by mariusor »

LukeShu wrote: post #65
gundmc wrote: post #22

Earlier quoted context omitted.

What do you mean by "barely" respecting robots.txt? Wouldn't that be more binary? Are they respecting some directives and ignoring others?

Amazonbot doesn't respect the `Crawl-Delay` directive. To be fair, Crawl-Delay is non-standard, but it is claimed to be respected by the other 3 most aggressive crawlers I see. And how often does it check robots.txt? ClaudeBot will make hundreds of thousands of requests before it re-checks robots.txt to see that you asked it to please stop DDoSing you.

One would think they'd at least respect the cache-control directives. Those have been in the web standards since forever.

Re: Amazon's AI crawler is making my Git server unstable

#224

Post by amanaplanacanal »

thayne wrote: post #99
pests wrote: post #74

Earlier quoted context omitted.

> `robots.txt` is the universal standard Quite the assumption, you just upset a bunch of alien species.

Universal within the scope of the Internet.

At least I don't think there are aliens connected...

Re: Amazon's AI crawler is making my Git server unstable

#225

Post by Daril »

I recently experienced a similar situation on a client's small ecommerce site. After struggling for 4 days to find the cause of a strange excessive server load every afternoon, I discovered that a Woocommerce site was being hammered by several bots at the same time : AHrefs, Bitespider, MJ12Bot, SemrushBot, Claude, GPT, ... I set up a proper robots.txt file (the content doesn't change frequently, so indexing it every day doesn't make sense), but also set up proper rules in .htaccess to block all these user-agents. I also banned 20 or more of the IP addresses of these bots. The next step is to set up fail2ban to automatically ban them when they change IP or add additional IP addresses.

Re: Amazon's AI crawler is making my Git server unstable

#226

Post by amanaplanacanal »

tepidsaucer wrote: post #209
advael wrote: post #145

Unless we start chopping these tech companies down there's not much hope for the public internet. They now have an incentive to crawl anything they can and have vastly more resources than even most governments. Most resources I need to host in a way that's internet facing are behind keyauth and I'm not sure I see a way around doing that for at least a while

That is an unworkable solution considering most of the people here are employed by these companies.

I would bet only a tiny minority here are working for big tech.

Re: Amazon's AI crawler is making my Git server unstable

#227

Post by sigmoid10 »

Aurornis wrote: post #159
Animats wrote: post #132

It's time for a lawyer letter. See the Computer Fraud and Abuse Act prosecution guidelines.[1] In general, the US Justice Department will not consider any access to open servers that's not clearly an attack to be "unauthorized access". But, "However, when authorizers later expressly revoke authorization—for example, through unambiguous written cease and desist communications that defendants receive and understand—the…

> Then you wait and see if Amazon stops. That’s if the requests are actually coming from Amazon, which seems very unlikely given some of the details in the post (rotating user agents, residential IPs, seemingly not interpreting robots.txt). The Amazon bot should come from known Amazon IP ranges and respect robots.txt. An Amazon engineer confirmed it in another comment: https://news.ycombinator.com/item?id=42751729 Th…

I wonder how the author hasn't reached this conclusion. The official Amazon Crawler docs literally tell you how to distinguish between legit Amazonbots and malicious copycats via DNS lookup: https://developer.amazon.com/amazonbot

Re: Amazon's AI crawler is making my Git server unstable

#229

Post by petee »

xena wrote: post #180
petee wrote: post #178

Earlier quoted context omitted.

Reverse DNS doesn't mean much, they can set it to anything; can you forward match them to any amazon domain?

It's forward confirming reverse DNS. I assumed that everyone does that by default.

What everyone does by default doesn't matter really here, it's that an IP owner/user can literally set the reverse to any arbitrary domain regardless if the actual domain has a record for that IP. What matters is both match, thats all I meant

Re: Amazon's AI crawler is making my Git server unstable

#230

Post by danaris »

WhyNotHugo wrote: post #221
Reason077 wrote: post #217

Earlier quoted context omitted.

Do we need a "robots must respect robots.txt" law?

If we did, bot authors would comply by just changing their User-Agent to something different that’s not expressly forbidden. (Disallowing * isn’t usually an option since it makes you disappear from search engines).

Any such law would absolutely have to include a requirement that all bots use at least some common element in their User-Agent strings to identify themselves as bots.
Post reply on HN