Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon's AI crawler is making my Git server unstable

xeiaso.net

Post reply on HN
Print view View original

161–170 of 261 posts

Re: Amazon's AI crawler is making my Git server unstable

#161

Post by Aurornis »

AyyEye wrote: post #137
Aurornis wrote: post #54

I don’t think I’d assume this is actually Amazon. The author is seeing requests from rotating residential IPs and changing user agent strings > It's futile to block AI crawler bots because they lie, change their user agent, use residential IP addresses as proxies, and more. Impersonating crawlers from big companies is a common technique for people trying to blend in. The fact that requests are coming from residential…

> The author is seeing requests from rotating residential IPs and changing user agent strings This type of thing is commercially available as a service[1]. Hundreds of Millions of networks backdoored and used as crawlers/scrapers because of an included library somewhere -- and ostensibly legal because somewhere in some ToS they had some generic line that could plausibly be extended to using you as a patsy for quasi-l…

Yes, we know, but the accusation is that Amazon is the source of the traffic.

If the traffic is coming from residential IPs then it’s most likely someone using these services and putting “AmazonBot” as a user agent to trick people.

Re: Amazon's AI crawler is making my Git server unstable

#162

Post by emmelaich »

trebor wrote: post #3

Upvoted because we’re seeing the same behavior from all AI and Seo bots. They’re BARELY respecting Robots.txt, and hard to block. And when they crawl, they spam and drive up load so high they crash many servers for our clients. If AI crawlers want access they can either behave, or pay. The consequence will almost universal blocks otherwise!

If they're AI bots it might be fun to feed them nonsense. Just send hack megabytes of "Bezos is a bozo" or something like that. Even more fun if you could cooperate with many other otherwise-unrelated websites, e.g. via time settings in a modified tarpit.

Re: Amazon's AI crawler is making my Git server unstable

#163

Post by [deleted] »

baobun wrote: post #82
Aurornis wrote: post #59

Earlier quoted context omitted.

Yes, but the point is that big company crawlers aren’t paying for questionably sourced residential proxies. If this person is seeing a lot of traffic from residential IPs then I would be shocked if it’s really Amazon. I think someone else is doing something sketchy and they put “AmazonBot” in the user agent to make victims think it’s Amazon. You can set the user agent string to anything you want, as we all know.

> Yes, but the point is that big company crawlers aren’t paying for questionably sourced residential proxies You'd be surprised...

[deleted]

Re: Amazon's AI crawler is making my Git server unstable

#165

Post by prinny_ »

SR2Z wrote: post #136
joecool1029 wrote: post #96

Earlier quoted context omitted.

Even some non-profit ignore it now, Internet Archive stopped respecting it years ago: https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...

IA actually has technical and moral reasons to ignore robots.txt. Namely, they want to circumvent this stuff because their goal is to archive EVERYTHING.

Isn’t this a weak argument? OpenAI could also say their goal is to learn everything, feed it to AI, advance humanity etc etc.

Re: Amazon's AI crawler is making my Git server unstable

#167

Post by [deleted] »

Aurornis wrote: post #59
OptionOfT wrote: post #49

Earlier quoted context omitted.

You should check your websites like grass dot io (I refuse to give them traffic). They pay you for your bandwidth while they resell it to 3rd parties, which is why a lot of bot traffic looks like it comes from residential IPs.

Yes, but the point is that big company crawlers aren’t paying for questionably sourced residential proxies. If this person is seeing a lot of traffic from residential IPs then I would be shocked if it’s really Amazon. I think someone else is doing something sketchy and they put “AmazonBot” in the user agent to make victims think it’s Amazon. You can set the user agent string to anything you want, as we all know.

[deleted]

Re: Amazon's AI crawler is making my Git server unstable

#168

Post by jazzyjackson »

baobun wrote: post #75
Ndymium wrote: post #8

I had this same issue recently. My Forgejo instance started to use 100 % of my home server's CPU as Claude and its AI friends from Meta and Google were hitting the basically infinite links at a high rate. I managed to curtail it with robots.txt and a user agent based blocklist in Caddy, but who knows how long that will work. Whatever happened to courtesy in scraping?

Mind sharing a decent robots.txt and/or user-agent list to block the AI crawlers?

Linked upthread: https://github.com/ai-robots-txt/ai.robots.txt/blob/main/rob...

Re: Amazon's AI crawler is making my Git server unstable

#170

Post by tredre3 »

ksec wrote: post #52
trebor wrote: post #3

Upvoted because we’re seeing the same behavior from all AI and Seo bots. They’re BARELY respecting Robots.txt, and hard to block. And when they crawl, they spam and drive up load so high they crash many servers for our clients. If AI crawlers want access they can either behave, or pay. The consequence will almost universal blocks otherwise!

Is there some way website can sell those Data to AI bot in a large zip file rather than being constantly DDoS? Or they could at least have the curtesy to scrap during night time / off peak hours.

There is project whose goal is to avoid this crawling-induced DDoS by maintaining a single web index: https://commoncrawl.org/
Post reply on HN