Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon's AI crawler is making my Git server unstable

xeiaso.net

Post reply on HN
Print view View original

101–110 of 261 posts

Re: Amazon's AI crawler is making my Git server unstable

#101

Post by to11mtm »

armchairhacker wrote: post #70

I like the solution in this comment: https://news.ycombinator.com/item?id=42727510 . Put a link somewhere in your site that no human would visit, disallow it in robots.txt (under a wildcard because apparently OpenAI’s crawler specifically ignores wildcards), and when an IP address visits the link ban it for 24 hours.

Uggh, web crawlers...

8ish years ago, at the shop I worked at we had a server taken down. It was an image server for vehicles. How did it go down? Well, the crawler in question somehow had access to vehicle image links we had due to our business. Unfortunately, the perfect storm of the image not actually existing (can't remember why, mighta been one of those weird cases where we did a re-inspection without issuing new inspection ID) resulted in them essentially DOSing our condition report image server. Worse, there was a bug in the error handler somehow, such that the server process restarted when this condition happened. This had the -additional- disadvantage of invalidating our 'for .NET 2.0, pretty dang decent' caching implementation...

It comes to mind because, I'm pretty sure we started doing some canary techniques just to be safe (Ironically, doing some simple ones were still cheaper than even adding a different web server.... yes we also fixed the caching issue... yes we also added a way to 'scream' if we got too many bad requests on that service.)

Re: Amazon's AI crawler is making my Git server unstable

#103

Post by aaomidi »

Szpadel wrote: post #92
armchairhacker wrote: post #70

I like the solution in this comment: https://news.ycombinator.com/item?id=42727510 . Put a link somewhere in your site that no human would visit, disallow it in robots.txt (under a wildcard because apparently OpenAI’s crawler specifically ignores wildcards), and when an IP address visits the link ban it for 24 hours.

I had to deal with some bot activities that used huge address space, and I tried something very similar, when condition confirming bot was detected I banned that IP for 24h but due to amount of IPs involved this did not have any impact on about if traffic my suggestion is to look very closely on headers that you receive (varnishlog in very nice of this and of you stare long enough at then you might stop something tha…

Maybe ban ASNs /s

Re: Amazon's AI crawler is making my Git server unstable

#104

Post by to11mtm »

Ndymium wrote: post #8

I had this same issue recently. My Forgejo instance started to use 100 % of my home server's CPU as Claude and its AI friends from Meta and Google were hitting the basically infinite links at a high rate. I managed to curtail it with robots.txt and a user agent based blocklist in Caddy, but who knows how long that will work. Whatever happened to courtesy in scraping?

> Whatever happened to courtesy in scraping?

When various companies got signal that at least for now they have a huge overton window of what is acceptable for AI to ingest, they are going to take all they can before regulation even tries to clamp down.

The bigger danger, is that one of these companies even (or, especially) one that claims to be 'Open', does so but gets to the point of being considered 'too big to fail' from an economic/natsec interest...

Re: Amazon's AI crawler is making my Git server unstable

#105

Post by to11mtm »

WarOnPrivacy wrote: post #94
dafelst wrote: post #84

Earlier quoted context omitted.

I worked for Microsoft doing malware detection back 10+ years ago, and questionably sourced proxies were well and truly on the table

>> but the point is that big company crawlers aren’t paying for questionably sourced residential proxies. > I worked for Microsoft doing malware detection back 10+ years ago, and questionably sourced proxies were well and truly on the table Big Company Crawlers using questionably sourced proxies - this seems striking. What can you share about it?

they probably can't because some of the proxies were used by TLAs is my guess...

Re: Amazon's AI crawler is making my Git server unstable

#106

Post by freetanga »

Probably dumb question, but any enlightenment would be welcome to help me learn:

Could this be prevented by having a link that when followed would serve a dynamically generated page that does all of the following:

A) insert some fake content outlining the oligarcs more lurid rumours or whichever disinformation you choose to push

C) embed links to assets in oligarchs companies so they get hit with some bandwith

C) dynamically create new Random pages that link to itself

And thus create an infinite loop, similar to a gzip bomb, which could potentially taint the model if done by enough people.

Re: Amazon's AI crawler is making my Git server unstable

#107

Post by to11mtm »

thayne wrote: post #80
neilv wrote: post #23

Can demonstrable ignoring of robots.txt help the cases of copyright infringement lawsuits against the "AI" companies, their partners, and customers?

Probably not copyright infringement. But it is probably (hopefully?) a violation of CFAA, both because it is effectively DDoSing you, and they are ignoring robots.txt. Maybe worth contacting law enforcement? Although it might not actually be Amazon.

Big thing worth asking here. Depending on what 'amazon' means here (i.e. known to be Amazon specific IPs vs Cloud IPs) it could just be someone running a crawler on AWS.

Or, folks failing the 'shared security model' of AWS and their stuff is compromised with botnets running on AWS.

Or, folks that are quasi-spoofing 'AmazonBot' because they think it will have a better not-block rate than anonymous or other requests...

Re: Amazon's AI crawler is making my Git server unstable

#109

Post by ADeerAppeared »

herpdyderp wrote: post #43
trebor wrote: post #3

Upvoted because we’re seeing the same behavior from all AI and Seo bots. They’re BARELY respecting Robots.txt, and hard to block. And when they crawl, they spam and drive up load so high they crash many servers for our clients. If AI crawlers want access they can either behave, or pay. The consequence will almost universal blocks otherwise!

> The consequence will almost universal blocks otherwise! How? The difficulty of doing that is the problem, isn't it? (Otherwise we'd just be doing that already.)

> (Otherwise we'd just be doing that already.)

Not quite what the original commenter meant but: WE ARE.

A major consequence of this reckless AI scraping is that it turbocharged the move away from the web and into closed ecosystems like Discord. Away from the prying eyes of most AI scrapers ... and the search engine indexes that made the internet so useful as an information resource.

Lots of old websites & forums are going offline as their hosts either cannot cope with the load or send a sizeable bill to the webmaster who then pulls the plug.

Post reply on HN