Amazon's AI crawler is making my Git server unstable

Post by DrBenCarson » Sat, Jan 18, 2025, 10:07 PM UTC

xena wrote: post #133

Animats wrote: post #132

It's time for a lawyer letter. See the Computer Fraud and Abuse Act prosecution guidelines.[1] In general, the US Justice Department will not consider any access to open servers that's not clearly an attack to be "unauthorized access". But, "However, when authorizers later expressly revoke authorization—for example, through unambiguous written cease and desist communications that defendants receive and understand—the…

Honestly, I figure that being on the front page of Hacker News like this is more than shame enough to get a human from the common sense department to read and respond to the email I sent politely asking them to stop scraping my git server. If I don't get a response by next Tuesday, I'm getting a lawyer to write a formal cease and desist letter.

Post by js4ever » Sat, Jan 18, 2025, 10:14 PM UTC

superjan wrote: post #130

Szpadel wrote: post #92

Earlier quoted context omitted.

I had to deal with some bot activities that used huge address space, and I tried something very similar, when condition confirming bot was detected I banned that IP for 24h but due to amount of IPs involved this did not have any impact on about if traffic my suggestion is to look very closely on headers that you receive (varnishlog in very nice of this and of you stare long enough at then you might stop something tha…

Why work hard… Train a model to recognize the AI bots!

Post by koito17 » Sat, Jan 18, 2025, 10:20 PM UTC

aaomidi wrote: post #103

Szpadel wrote: post #92

Earlier quoted context omitted.

I had to deal with some bot activities that used huge address space, and I tried something very similar, when condition confirming bot was detected I banned that IP for 24h but due to amount of IPs involved this did not have any impact on about if traffic my suggestion is to look very closely on headers that you receive (varnishlog in very nice of this and of you stare long enough at then you might stop something tha…

Maybe ban ASNs /s

Post by thayne » Sat, Jan 18, 2025, 10:31 PM UTC

to11mtm wrote: post #107

thayne wrote: post #80

Earlier quoted context omitted.

Probably not copyright infringement. But it is probably (hopefully?) a violation of CFAA, both because it is effectively DDoSing you, and they are ignoring robots.txt. Maybe worth contacting law enforcement? Although it might not actually be Amazon.

Big thing worth asking here. Depending on what 'amazon' means here (i.e. known to be Amazon specific IPs vs Cloud IPs) it could just be someone running a crawler on AWS. Or, folks failing the 'shared security model' of AWS and their stuff is compromised with botnets running on AWS. Or, folks that are quasi-spoofing 'AmazonBot' because they think it will have a better not-block rate than anonymous or other requests...

Post by bb010g » Sat, Jan 18, 2025, 10:41 PM UTC

shakna wrote: post #141

armchairhacker wrote: post #70

I like the solution in this comment: https://news.ycombinator.com/item?id=42727510 . Put a link somewhere in your site that no human would visit, disallow it in robots.txt (under a wildcard because apparently OpenAI’s crawler specifically ignores wildcards), and when an IP address visits the link ban it for 24 hours.

When I was writing a crawler for my search engine (now offline), I found almost no crawler library actually compliant with the real world. So I ended up going to a lot of effort to write one that complied with Amazon and Google's rather complicated nested robots files, including respecting the cool off periods as requested. ... And then found their own crawlers can't parse their own manifests.