Sure it wasn't a AI crawler?
There's no evidence the crawlers are AI-related.
Understanding the recent DDoS attack against Read the Docs
81–90 of 91 posts
Re: Understanding the recent DDoS attack against Read the Docs
#82A more interesting question is, what exactly do the attackers gain from hitting read the docs? Most of their docs hosting is static/easily CDN cached. Unlike database bound sites, you would need a lot more traffic to overload pure/mostly static hosting. Maybe it's a malicious AI lab looking to deny their competitors training data? As far as infosec profiling goes, this is probably the oddest case I have heard of. I a…
Author here. This was not a misconfigured data scraper. We see those every week[1]. This attack wasn't scraping useful content. It was almost entirely 404s and 302s and pulled virtually zero real docs. It specifically looked for URLs not served by the CDN and when it found a pattern, did millions of variations of it. Whether built by an AI or not, it was designed to cause outages and financial damage from autoscaling…
Do you have a list of the addresses that were hammering your site? Have you tried any of the techniques I list here? [1] Do sets of the IP's show up in here [2]? Are the bots mostly residential, VPS, Tor? What is the HTTP protocol breakdown? HTTP/1.1, 2.0, 3.0? Are they missing any expected client headers? Have you tried blackhole routing any of them from an out of band management console?
# only useful if not behind a CDN
for Ip in $(cat /dev/shm/list-of-attackers.ipset);do ip route add blackhole "${Ip}" 2>/dev/null;done
[1] - https://nochan.net/b/Internet-Crap/20260606-How-To-Block-Som...
Re: Understanding the recent DDoS attack against Read the Docs
#83Earlier quoted context omitted.
Author here. This was not a misconfigured data scraper. We see those every week[1]. This attack wasn't scraping useful content. It was almost entirely 404s and 302s and pulled virtually zero real docs. It specifically looked for URLs not served by the CDN and when it found a pattern, did millions of variations of it. Whether built by an AI or not, it was designed to cause outages and financial damage from autoscaling…
Do you have a list of the addresses that were hammering your site? Have you tried any of the techniques I list here? [1] Do sets of the IP's show up in here [2]? Are the bots mostly residential, VPS, Tor? What is the HTTP protocol breakdown? HTTP/1.1, 2.0, 3.0? Are they missing any expected client headers? Have you tried blackhole routing any of them from an out of band management console? # only useful if not behind…
> Do you have a list of the addresses that were hammering your site?
I could get a list of IPs, but it wouldn't be particularly helpful. It was hundreds of millions of unique IPs. Most IPs made fewer than 100 requests and then disappeared.
> Have you tried any of the techniques I list here?
We were already doing essentially all of the techniques there and they were not helpful in this attack. This wasn't some spider run amok. This was a dedicated attack with intelligence behind it.
> Do sets of the IP's show up in here [2]?
If we're looking at individual IPs, we've already lost.
> Are the bots mostly residential, VPS, Tor? What is the HTTP protocol breakdown? HTTP/1.1, 2.0, 3.0? Are they missing any expected client headers?
The article outlines most of this. It was a mix of residential and major/minor hosting providers and some corporates. If I were to say which was the largest, it was minor hosting providers. However, the attack pivoted between coming from different sources. At first it was coming mostly from minor hosting providers/VPS. By the end, it was entirely residential and corporate IPs. Tor was not involved at all (RTD supports accessing through Tor, but you are more likely to get a challenge). All the expected headers were there and UAs cycled between a very large set of standard UAs for normal browsers/devices. The attack was 95%+ HTTP2/HTTP3.
> [Edit] appears you are behind Cloudflare so the blackhole would be up to them. One could still return a 429 or 525 to the attackers.
It's trivial to setup an IP list[1] in CF and then you can apply all sorts of rules (lower rate limits, outright challenges, etc.) to it. It can be managed through Terraform as well. You can also use CIDR notation. This along with more classifications to specific ASNs are something we are looking at.
[1] https://developers.cloudflare.com/waf/tools/lists/custom-lis...
Re: Understanding the recent DDoS attack against Read the Docs
#84Earlier quoted context omitted.
Do you have a list of the addresses that were hammering your site? Have you tried any of the techniques I list here? [1] Do sets of the IP's show up in here [2]? Are the bots mostly residential, VPS, Tor? What is the HTTP protocol breakdown? HTTP/1.1, 2.0, 3.0? Are they missing any expected client headers? Have you tried blackhole routing any of them from an out of band management console? # only useful if not behind…
> Do you have a list of the addresses that were hammering your site? I could get a list of IPs, but it wouldn't be particularly helpful. It was hundreds of millions of unique IPs. Most IPs made fewer than 100 requests and then disappeared. > Have you tried any of the techniques I list here? We were already doing essentially all of the techniques there and they were not helpful in this attack. This wasn't some spider…
Inside this .bz2 [1] there is a directory called bh_routes/bh_vps_120/ and in that there are many AS files named for many of the VPS/Server providers. If you site does not need inbound connections from such places that is a starting point to reduce the traffic a bit. Each file has a comment with the AS#/whois name if you want to double check who/what you may be blocking. Not perfect but it may make a dent in the traffic. Or ask Cloudflare if they have an option to block anything that is not residential or LTE wireless. Even if you want to allow bots, maybe have a "shields up" mode where bots are sent away until the attack passes.
Another potential option would be to configure nginx IP limits using the Cloudflare header that represents the IP, set the shared memory size rather high and return 525 to the IP's exceeding a limit to avoid them hitting your redirect rules though it feels like CF should be able to create something custom for this assuming its a paid account.
Any IP that is not VPS/server should be archived for the feds as they are working on shutting down residential proxy providers including apps that are turning peoples cell phones into proxies.
[1] - https://nochan.net/b/Internet-Crap/20260606-How-To-Block-Som...
Re: Understanding the recent DDoS attack against Read the Docs
#85Earlier quoted context omitted.
I don't quite understand your post, but is your question why don't the ISPs of the sources of the abusive traffic sort it out? The distributed nature of DDoS means each participating host isn't sending that much traffic, and there are often tens or hundreds of thousands of participating hosts. An ISP should verify claims of abuse before cutting off customers, and since most of the customers are presumably unaware of…
> most attacks were 90 seconds long ... there's no way I'm convincing an ISP to drop a pwned customer over that. Every victim (such as readthedocs), or even people sharing blocklists to avoid becoming a victim, blocking that ISP's ranges until they do clean up their network could be a convincing argument? As you say, even at 90 seconds, it's clear to all involved parties that the customer is pwned or malicious. Such…
www.uceprotect.net does that for email. It's a DNS blacklist that puts hole networks on it, even if "only" a individual hosts SPAMs.
It's a double-edged sword. That is how you end up with most ISPs blocking port 25 completely. If your hosting provider is on the list you are collateral damage. You yourself can do very little to remedy the situation except to beg your provider "to look into it".
What do you expect the ISPs to do in this story? We are talking about TSL connections. Block port 80 and 443 and expect the costumers to use your HTTPS-Proxy. Than they could inspect and block individual actions.
Re: Understanding the recent DDoS attack against Read the Docs
#86Earlier quoted context omitted.
> most attacks were 90 seconds long ... there's no way I'm convincing an ISP to drop a pwned customer over that. Every victim (such as readthedocs), or even people sharing blocklists to avoid becoming a victim, blocking that ISP's ranges until they do clean up their network could be a convincing argument? As you say, even at 90 seconds, it's clear to all involved parties that the customer is pwned or malicious. Such…
> Every victim (such as readthedocs), or even people sharing blocklists The report makes it pretty clear that the attack was distributed enough that profiling for blocklists was ineffective.
Whose IPs were they using then? All clean IP addresses from ISPs who are responsive to abuse reports? The post says no such thing
Re: Understanding the recent DDoS attack against Read the Docs
#87Earlier quoted context omitted.
> most attacks were 90 seconds long ... there's no way I'm convincing an ISP to drop a pwned customer over that. Every victim (such as readthedocs), or even people sharing blocklists to avoid becoming a victim, blocking that ISP's ranges until they do clean up their network could be a convincing argument? As you say, even at 90 seconds, it's clear to all involved parties that the customer is pwned or malicious. Such…
> As you say, even at 90 seconds, it's clear to all involved parties that the customer is pwned or malicious. It sure is --- but an ISP would want to observe the traffic themselves, and if it's a 90 second attack every so often, chances are they won't see it when they look. When it's volumetric reflection, you can probably tell them how to send a request and see the response, and maybe they'll contact the customer, b…
> an ISP would want to observe the traffic themselves
My ISP didn't, when they got access logs from a service I attacked as a teenager and asked me to explain that to get the connection unblocked
Idk, at the moment we're simply not even trying to set a standard. Maybe it would work reasonably well when the ISP needs to observe the traffic and, after a few days of the initial report, they observe a netflow that matches a new abuse report. Even if we set low standards, currently, too few people are sending abuse notifications instead of just sticking it behind the great internet vetting service and calling it good
> maybe they'll contact the customer, but maybe they'll just sit on it.
That's the core point no? If they don't care about their abusive traffic, nullroute their ranges. If admins consistently do that, the abuse has to stop or the ISP goes out of business
Re: Understanding the recent DDoS attack against Read the Docs
#88Earlier quoted context omitted.
> most attacks were 90 seconds long ... there's no way I'm convincing an ISP to drop a pwned customer over that. Every victim (such as readthedocs), or even people sharing blocklists to avoid becoming a victim, blocking that ISP's ranges until they do clean up their network could be a convincing argument? As you say, even at 90 seconds, it's clear to all involved parties that the customer is pwned or malicious. Such…
www.uceprotect.net does that for email. It's a DNS blacklist that puts hole networks on it, even if "only" a individual hosts SPAMs. It's a double-edged sword. That is how you end up with most ISPs blocking port 25 completely. If your hosting provider is on the list you are collateral damage. You yourself can do very little to remedy the situation except to beg your provider "to look into it". What do you expect the…
> If your hosting provider is on the list you are collateral damage.
Yes, and this sucks. I moved ISPs because the original one had burned IP addresses that you can't send email from. Every ISP that gets the ranges burned like that will eventually either goes out of business or gets their act together. Not by tomorrow, but eventually
Like, the only other alternative outcome I see is that everyone has to pass through a central surveillance point that decides who's benign and who's naughty, and since nobody wants that... what else are we to do but report abuse?
> What do you expect the ISPs to do in this story? We are talking about TLS connections.
What they've done to me when I abused a service as a teenager, the ISP got an abuse notification: cut off the connection, ask the subscriber wtf this traffic is and how they're going to make sure it doesn't happen again (at which point my dad, the subscriber, came to me and asked if I knew something about this.... yeah ^^')
TLS doesn't matter because the ones receiving the abusive traffic can say what it was. Their access logs will contain the decrypted information and that should be put in an abuse report. If the customer denies everything, turn on netflow logging for a month and see if a future abuse report comes in that can be correlated against the netflow logs. Or have netflow logs stored for 1h by default and retain the entries for which an abuse notification came in (grep for IP addresses in incoming notifications). Lots of reasonable options there; this isn't the difficulty. It's getting people to send that abuse notification so the ISP can identify the problematic subscribers
Re: Understanding the recent DDoS attack against Read the Docs
#89Earlier quoted context omitted.
> most attacks were 90 seconds long ... there's no way I'm convincing an ISP to drop a pwned customer over that. Every victim (such as readthedocs), or even people sharing blocklists to avoid becoming a victim, blocking that ISP's ranges until they do clean up their network could be a convincing argument? As you say, even at 90 seconds, it's clear to all involved parties that the customer is pwned or malicious. Such…
www.uceprotect.net does that for email. It's a DNS blacklist that puts hole networks on it, even if "only" a individual hosts SPAMs. It's a double-edged sword. That is how you end up with most ISPs blocking port 25 completely. If your hosting provider is on the list you are collateral damage. You yourself can do very little to remedy the situation except to beg your provider "to look into it". What do you expect the…
[flagged]
Re: Understanding the recent DDoS attack against Read the Docs
#90Earlier quoted context omitted.
Do you have a list of the addresses that were hammering your site? Have you tried any of the techniques I list here? [1] Do sets of the IP's show up in here [2]? Are the bots mostly residential, VPS, Tor? What is the HTTP protocol breakdown? HTTP/1.1, 2.0, 3.0? Are they missing any expected client headers? Have you tried blackhole routing any of them from an out of band management console? # only useful if not behind…
> Do you have a list of the addresses that were hammering your site? I could get a list of IPs, but it wouldn't be particularly helpful. It was hundreds of millions of unique IPs. Most IPs made fewer than 100 requests and then disappeared. > Have you tried any of the techniques I list here? We were already doing essentially all of the techniques there and they were not helpful in this attack. This wasn't some spider…
[flagged]