Back to Gopher. They'll never get us there!
Amazon's AI crawler is making my Git server unstable
111–120 of 261 posts
Re: Amazon's AI crawler is making my Git server unstable
#112Earlier quoted context omitted.
> `robots.txt` is the universal standard Quite the assumption, you just upset a bunch of alien species.
Dammit. Unchecked geocentric model privilege, sorry about that.
I mean it might just be a matter of the right UK person filing a case; I suppose my main understanding is UK Libel/Slander laws but if my US brain goes with that my head says the burden of proof is on non-infringement.
(But again, I don't know UK law.)
Re: Amazon's AI crawler is making my Git server unstable
#113[flagged]
HN isnt a monolith
Re: Amazon's AI crawler is making my Git server unstable
#114I’m surprised everyone else’s servers are struggling to handle a couple of bot scrapes. I run a couple of public facing websites on a NUC and it just… chugs along? This is also amidst the constant barrage of OSINT attempts at my IP.
Seems some of these bots are behaving abusively on sites with lots of links (like git forges). I have some sites receiving 200 requests per day and some receiving 1 million requests per day from these AI bots, depending on the design of the site.
Re: Amazon's AI crawler is making my Git server unstable
#115I don’t think I’d assume this is actually Amazon. The author is seeing requests from rotating residential IPs and changing user agent strings > It's futile to block AI crawler bots because they lie, change their user agent, use residential IP addresses as proxies, and more. Impersonating crawlers from big companies is a common technique for people trying to blend in. The fact that requests are coming from residential…
I wouldn't put it past any company these days doing crawling in an aggressive manner to use proxy networks.
With the amount of "if cloud IP then block" rules in place for many things (to weed out streaming VPNs and "potential" ddos-ing) I wouldn't doubt that at all.
Re: Amazon's AI crawler is making my Git server unstable
#116I’m surprised everyone else’s servers are struggling to handle a couple of bot scrapes. I run a couple of public facing websites on a NUC and it just… chugs along? This is also amidst the constant barrage of OSINT attempts at my IP.
Gitea in particular is a worst case for this. Gitea shows details about every file at every version and every commit if you click enough. The bots click every link. This fixed cost adds up when hundreds of IPs are at different levels of clicking of every link.
Re: Amazon's AI crawler is making my Git server unstable
#117Return "402 Payment Required" and block?
No. Feed them shit. Code with deliberate security vulns and so on.
Re: Amazon's AI crawler is making my Git server unstable
#118The best way to fight this would not to block them, that
does not cause Amazon/others anything. (clearly).
What if instead it was possible to feed the bots clearly damaging and harmfull content?
If done on a larger scale, and Amazon discovers the poisoned pills they could have to spend money rooting it out, quick like, and make attempts to stop their bots to ingest it.
Of course nobody wants to have that tuff on their own site though. That is the biggest problem with this.
Re: Amazon's AI crawler is making my Git server unstable
#119He seems to have a mistake in his rule? He's got "(Amazon)" while Amazon lists their useragent as "(Amazonbot/0.1;"
The author's pronouns can be found here: https://github.com/Xe
Re: Amazon's AI crawler is making my Git server unstable
#120Earlier quoted context omitted.
Dammit. Unchecked geocentric model privilege, sorry about that.
I mean it might just be a matter of the right UK person filing a case; I suppose my main understanding is UK Libel/Slander laws but if my US brain goes with that my head says the burden of proof is on non-infringement. (But again, I don't know UK law.)
[deleted]