Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon's AI crawler is making my Git server unstable

xeiaso.net

Post reply on HN
Print view View original

41–50 of 261 posts

Re: Amazon's AI crawler is making my Git server unstable

#41

Post by 23B1 »

Analemma_ wrote: post #34
23B1 wrote: post #26

[flagged]

Most of those of those artists aren’t any better though. I’m on a couple artists’ forums and outlets like Tumblr, and I saw firsthand the immediate, total 180 re: IP protection when genAI showed up. Overnight, everybody went from “copying isn’t theft, it leaves the original!” and other such mantras, to being die-hard IP maximalists. To say nothing of how they went from “anything can be art and it doesn’t matter what…

Manga nerds on Tumblr aren't the artists I'm worried about. I'm talking about people whose intellectual labor is being laundered by gigacorps and the inane defenses mounted by their techbro serfdom.

Re: Amazon's AI crawler is making my Git server unstable

#43

Post by herpdyderp »

trebor wrote: post #3

Upvoted because we’re seeing the same behavior from all AI and Seo bots. They’re BARELY respecting Robots.txt, and hard to block. And when they crawl, they spam and drive up load so high they crash many servers for our clients. If AI crawlers want access they can either behave, or pay. The consequence will almost universal blocks otherwise!

> The consequence will almost universal blocks otherwise!

How? The difficulty of doing that is the problem, isn't it? (Otherwise we'd just be doing that already.)

Re: Amazon's AI crawler is making my Git server unstable

#46

Post by bwfan123 »

jsheard wrote: post #13
Ndymium wrote: post #8

I had this same issue recently. My Forgejo instance started to use 100 % of my home server's CPU as Claude and its AI friends from Meta and Google were hitting the basically infinite links at a high rate. I managed to curtail it with robots.txt and a user agent based blocklist in Caddy, but who knows how long that will work. Whatever happened to courtesy in scraping?

> Whatever happened to courtesy in scraping? Money happened. AI companies are financially incentivized to take as much data as possible, as quickly as possible, from anywhere they can get it, and for now they have so much cash to burn that they don't really need to be efficient about it.

not only money, but also a culture of "all your data belong to us" because our ai going to save you and the world.

the hubris reminds me of dot-com era. that bust left a huge wreckage. not sure how this one is going to land.

Re: Amazon's AI crawler is making my Git server unstable

#48

Post by jks »

mschuster91 wrote: post #12
trebor wrote: post #3

Upvoted because we’re seeing the same behavior from all AI and Seo bots. They’re BARELY respecting Robots.txt, and hard to block. And when they crawl, they spam and drive up load so high they crash many servers for our clients. If AI crawlers want access they can either behave, or pay. The consequence will almost universal blocks otherwise!

Global tarpit is the solution. It makes sense anyway even without taking AI crawlers into account. Back when I had to implement that, I went the semi manual route - parse the access log and any IP address averaging more than X hits a second on /api gets a -j TARPIT with iptables [1]. Not sure how to implement it in the cloud though, never had the need for that there yet. [1] https://gist.github.com/flaviovs/103a0dbf6…

One such tarpit (Nepenthes) was just recently mentioned on Hacker News: https://news.ycombinator.com/item?id=42725147

Their site is down at the moment, but luckily they haven't stopped Wayback Machine from crawling it: https://web.archive.org/web/20250117030633/https://zadzmo.or...

Re: Amazon's AI crawler is making my Git server unstable

#49

Post by OptionOfT »

thayne wrote: post #35

Are you sure it isn't a DDoS masquerading as Amazon? Requests coming from residential ips is really suspicious. Edit: the motivation for such a DDoS might be targeting Amazon, by taking down smaller sites and making it look like amazon is responsible. If it is Amazon one place to start is blocking all the the ip ranges they publish. Although it sounds like there are requests outside those ranges...

You should check your websites like grass dot io (I refuse to give them traffic).

They pay you for your bandwidth while they resell it to 3rd parties, which is why a lot of bot traffic looks like it comes from residential IPs.

Re: Amazon's AI crawler is making my Git server unstable

#50

Post by xena »

jsheard wrote: post #33
Vampiero wrote: post #30

Earlier quoted context omitted.

> The consequence will almost universal blocks otherwise! Who cares? They've already scraped the content by then.

Bold to assume that an AI scraper won't come back to download everything again, just in case there's any new scraps of data to extract. OP mentioned in the other thread that this bot had pulled 3TB so far, and I doubt their git server actually has 3TB of unique data, so the bot is probably pulling the same data over and over again.

FWIW that includes other scrapers, Amazon's is just the one that showed up the most in the logs.
Post reply on HN