Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon's AI crawler is making my Git server unstable

xeiaso.net

Post reply on HN
Print view View original

181–190 of 261 posts

Re: Amazon's AI crawler is making my Git server unstable

#181

Post by [deleted] »

Aurornis wrote: post #159
Animats wrote: post #132

It's time for a lawyer letter. See the Computer Fraud and Abuse Act prosecution guidelines.[1] In general, the US Justice Department will not consider any access to open servers that's not clearly an attack to be "unauthorized access". But, "However, when authorizers later expressly revoke authorization—for example, through unambiguous written cease and desist communications that defendants receive and understand—the…

> Then you wait and see if Amazon stops. That’s if the requests are actually coming from Amazon, which seems very unlikely given some of the details in the post (rotating user agents, residential IPs, seemingly not interpreting robots.txt). The Amazon bot should come from known Amazon IP ranges and respect robots.txt. An Amazon engineer confirmed it in another comment: https://news.ycombinator.com/item?id=42751729 Th…

[deleted]

Re: Amazon's AI crawler is making my Git server unstable

#183

Post by andrethegiant »

subarctic wrote: post #179
andrethegiant wrote: post #177

I’m working on a centralized platform[1] to help web crawlers be polite by default by respecting robots.txt, 429s, etc, and sharing a platform-wide TTL cache just for crawlers. The goal is to reduce global bot traffic by providing a convenient option to crawler authors that makes their bots play nice with the open web. [1] https://crawlspace.dev

How does it compare to common crawl?

CommonCrawl’s data is up to a month old. Here you can write a custom crawler, and have a REST/RAG API that you can use with your apps and agents for the data that your crawler finds. All the while, it builds up a platform-wide cache, so duplicate/redundant requests don’t reach their origin servers.

Re: Amazon's AI crawler is making my Git server unstable

#185

Post by guardiangod »

Aurornis wrote: post #59
OptionOfT wrote: post #49

Earlier quoted context omitted.

You should check your websites like grass dot io (I refuse to give them traffic). They pay you for your bandwidth while they resell it to 3rd parties, which is why a lot of bot traffic looks like it comes from residential IPs.

Yes, but the point is that big company crawlers aren’t paying for questionably sourced residential proxies. If this person is seeing a lot of traffic from residential IPs then I would be shocked if it’s really Amazon. I think someone else is doing something sketchy and they put “AmazonBot” in the user agent to make victims think it’s Amazon. You can set the user agent string to anything you want, as we all know.

I used to work for malware detection for a security company, and we looked at residential IP proxy services.

They are very, very, very expensive for the amount of data you get. You are paying for per bit of data. Even with Amazon's money, the number quickly become untenable.

It was literally cheaper for us to subscribe to business ADSL/cable/fiber optic services to our corp office buildings and thrunk them together.

Re: Amazon's AI crawler is making my Git server unstable

#186

Post by guardiangod »

xena wrote: post #173
cmeacham98 wrote: post #134

Earlier quoted context omitted.

I work for Amazon, but not directly on web crawling. Based on the internal information I have been able to gather, it is highly unlikely this is actually Amazon. Amazonbot is supposed to respect robots.txt and should always come from an Amazon-owned IP address (You can see verification steps here: https://developer.amazon.com/en/amazonbot ). I've forwarded this internally just in case there is some crazy internal tea…

Randomly selected IPs from my logs show that 80% of them have the matching that forward confirming reverse DNS domain. The most aggressive ones were from the amazonbot domain. Believe what you want though. Search for `xeiaso.net` in ticketing if you want proof.

So you said the IPs are residential IP, but their reverse DNS points to a amazonbot domain? Does that even make sense?

Re: Amazon's AI crawler is making my Git server unstable

#187

Post by AnonC »

joecool1029 wrote: post #96
unsnap_biceps wrote: post #28

Earlier quoted context omitted.

I believe that a number of AI bots only respect robot.txt entries that explicitly define their static user agent name. They ignore wildcards in user agents. That counts as barely imho. I found this out after OpenAI was decimating my site and ignoring the wildcard deny all. I had to add entires specifically for their three bots to get them to stop.

Even some non-profit ignore it now, Internet Archive stopped respecting it years ago: https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...

As I recall, this is outdated information. Internet Archive does respect robots.txt and will remove a site from its archive based on robots.txt. I have done this a few years after your linked blog post to get an inconsequential site removed from archive.org.

Re: Amazon's AI crawler is making my Git server unstable

#188

Post by trod1234 »

Szpadel wrote: post #92
armchairhacker wrote: post #70

I like the solution in this comment: https://news.ycombinator.com/item?id=42727510 . Put a link somewhere in your site that no human would visit, disallow it in robots.txt (under a wildcard because apparently OpenAI’s crawler specifically ignores wildcards), and when an IP address visits the link ban it for 24 hours.

I had to deal with some bot activities that used huge address space, and I tried something very similar, when condition confirming bot was detected I banned that IP for 24h but due to amount of IPs involved this did not have any impact on about if traffic my suggestion is to look very closely on headers that you receive (varnishlog in very nice of this and of you stare long enough at then you might stop something tha…

If you just block the connection, you send a signal that you are blocking it, and they will change it. You need to impose cost per every connection through QoS buckets.

If they rotate IPs, ban by ASN, have a page with some randomized pseudo looking content in the source (not static), and explain that the traffic allocated to this ASN has exceed normal user limits and has been rate limited (to a crawl).

Have graduated responses starting at a 72 hour ban where every page thereafter regardless of URI results in that page and rate limit. Include a contact email address that is dynamically generated by bucket, and validate all inbound mail that it matches DMARC for Amazon. Be ready to provide a log of abusive IP addresses.

That way if amazon wants to take action, they can but its in their ballpark. You gatekeep what they can do on your site with your bandwidth. Letting them run hog wild and steal bandwidth from you programmatically is unacceptable.

Re: Amazon's AI crawler is making my Git server unstable

#189

Post by trod1234 »

superjan wrote: post #130
Szpadel wrote: post #92

Earlier quoted context omitted.

I had to deal with some bot activities that used huge address space, and I tried something very similar, when condition confirming bot was detected I banned that IP for 24h but due to amount of IPs involved this did not have any impact on about if traffic my suggestion is to look very closely on headers that you receive (varnishlog in very nice of this and of you stare long enough at then you might stop something tha…

Why work hard… Train a model to recognize the AI bots!

This isn't a problem domain that models are capable of solving.

Ultimately in two party communications, computers are mostly constrained by determinism, and the resulting halting/undecidability problems (in core computer science).

All AI Models are really bad at solving stochastic types of problems. They can approximate generally only to a point after which it falls off. Temporal consistency im time series data is also a major weakness. Throw the two together, and models can't really solve it. They can pattern match to a degree but that is the limit.

Re: Amazon's AI crawler is making my Git server unstable

#190

Post by pixl97 »

koito17 wrote: post #146
aaomidi wrote: post #103

Earlier quoted context omitted.

Maybe ban ASNs /s

This was indeed one mitigation used by a site to prevent bots hosted on AWS from uploading CSAM and generating bogus reports to the site's hosting provider.[1] In any case, I agree with the sarcasm. Blocking data center IPs may not help the OP, because some of the bots are resorting to residential IP addresses. [1] https://news.ycombinator.com/item?id=26865236

Ya if it's also coming from residences it's probably some kind of botnet
Post reply on HN