We have had the same problem at my client now for the last couple of months, but from Facebook (using their IP ranges). They don’t even respect the 429 headers and the business is hesitant to outright ban them in case it impacts open graph or Facebook advertising tooling.
Amazon's AI crawler is making my Git server unstable
91–100 of 261 posts
Re: Amazon's AI crawler is making my Git server unstable
#92I like the solution in this comment: https://news.ycombinator.com/item?id=42727510 . Put a link somewhere in your site that no human would visit, disallow it in robots.txt (under a wildcard because apparently OpenAI’s crawler specifically ignores wildcards), and when an IP address visits the link ban it for 24 hours.
I had to deal with some bot activities that used huge address space, and I tried something very similar, when condition confirming bot was detected I banned that IP for 24h
but due to amount of IPs involved this did not have any impact on about if traffic
my suggestion is to look very closely on headers that you receive (varnishlog in very nice of this and of you stare long enough at then you might stop something that all those requests have in common that would allow you to easily identify them (like very specific and usual combination of reported language and geo location, or the same outdated browser version, etc)
Re: Amazon's AI crawler is making my Git server unstable
#93He seems to have a mistake in his rule?
He's got "(Amazon)" while Amazon lists their useragent as "(Amazonbot/0.1;"
Re: Amazon's AI crawler is making my Git server unstable
#94Earlier quoted context omitted.
Yes, but the point is that big company crawlers aren’t paying for questionably sourced residential proxies. If this person is seeing a lot of traffic from residential IPs then I would be shocked if it’s really Amazon. I think someone else is doing something sketchy and they put “AmazonBot” in the user agent to make victims think it’s Amazon. You can set the user agent string to anything you want, as we all know.
I worked for Microsoft doing malware detection back 10+ years ago, and questionably sourced proxies were well and truly on the table
>> but the point is that big company crawlers aren’t paying for questionably sourced residential proxies.
> I worked for Microsoft doing malware detection back 10+ years ago, and questionably sourced proxies were well and truly on the table
Big Company Crawlers using questionably sourced proxies - this seems striking. What can you share about it?
Re: Amazon's AI crawler is making my Git server unstable
#95Earlier quoted context omitted.
Yes, but the point is that big company crawlers aren’t paying for questionably sourced residential proxies. If this person is seeing a lot of traffic from residential IPs then I would be shocked if it’s really Amazon. I think someone else is doing something sketchy and they put “AmazonBot” in the user agent to make victims think it’s Amazon. You can set the user agent string to anything you want, as we all know.
> Yes, but the point is that big company crawlers aren’t paying for questionably sourced residential proxies You'd be surprised...
>> Yes, but the point is that big company crawlers aren’t paying for questionably sourced residential proxies
> You'd be surprised...
Surprised by what? What do you know?
Re: Amazon's AI crawler is making my Git server unstable
#96Earlier quoted context omitted.
What do you mean by "barely" respecting robots.txt? Wouldn't that be more binary? Are they respecting some directives and ignoring others?
I believe that a number of AI bots only respect robot.txt entries that explicitly define their static user agent name. They ignore wildcards in user agents. That counts as barely imho. I found this out after OpenAI was decimating my site and ignoring the wildcard deny all. I had to add entires specifically for their three bots to get them to stop.
Even some non-profit ignore it now, Internet Archive stopped respecting it years ago: https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
Re: Amazon's AI crawler is making my Git server unstable
#97Earlier quoted context omitted.
What do you mean by "barely" respecting robots.txt? Wouldn't that be more binary? Are they respecting some directives and ignoring others?
I believe that a number of AI bots only respect robot.txt entries that explicitly define their static user agent name. They ignore wildcards in user agents. That counts as barely imho. I found this out after OpenAI was decimating my site and ignoring the wildcard deny all. I had to add entires specifically for their three bots to get them to stop.
[deleted]
Re: Amazon's AI crawler is making my Git server unstable
#98Earlier quoted context omitted.
> Whatever happened to courtesy in scraping? Money happened. AI companies are financially incentivized to take as much data as possible, as quickly as possible, from anywhere they can get it, and for now they have so much cash to burn that they don't really need to be efficient about it.
not only money, but also a culture of "all your data belong to us" because our ai going to save you and the world. the hubris reminds me of dot-com era. that bust left a huge wreckage. not sure how this one is going to land.
It's gonna be rough. If you can't make money charging people $200 a month for your service then something is deeply wrong.
Re: Amazon's AI crawler is making my Git server unstable
#99Earlier quoted context omitted.
In the UK, the Computer Misuse Act applies if: * There is knowledge that the intended access was unauthorised * There is an intention to secure access to any program or data held in a computer I imagine US law has similar definitions of unauthorized access? `robots.txt` is the universal standard for defining what is unauthorised access for bots. No programmer could argue they aren't aware of this, and ignoring it, fo…
> `robots.txt` is the universal standard Quite the assumption, you just upset a bunch of alien species.
Universal within the scope of the Internet.
Re: Amazon's AI crawler is making my Git server unstable
#100It seems like git self-hosters frequently encounter DDoS issues. I know it's not typical for free software, but I wonder if gating file contents behind a login and allowing registrations could be the answer for self-hosting repositories on the cheap.