Understanding the recent DDoS attack against Read the Docs
about.readthedocs.com
Understanding the recent DDoS attack against Read the Docs
1–10 of 73 posts
Re: Understanding the recent DDoS attack against Read the Docs
#2I'm curious if anybody could speculate who would be attacking a documentation silo, and to what end?
Re: Understanding the recent DDoS attack against Read the Docs
#3I'm curious if anybody could speculate who would be attacking a documentation silo, and to what end?
Could be testing in preparation for attacking something more critical?
Re: Understanding the recent DDoS attack against Read the Docs
#4I'm curious if anybody could speculate who would be attacking a documentation silo, and to what end?
I'm the author of the blog. I don't know. Internally, we were half joking that we were going to get ransom notice, but we never did.
The only thing that sort of correlates with this attack is that before it started, we began rolling out some slightly more aggressive rate limits one by one. This was mostly because anytime any new "company" thinks they're going to catchup with Claude/OpenAI, they scrape us very aggressively (and they're not respectful about it). My guess is that the attackers behind this attack were already probing us (they were) and they thought the window of opportunity might be closing.
Re: Understanding the recent DDoS attack against Read the Docs
#5Interesting that the Under Attack Mode wasn’t used at all here. I understand not wanting to break APIs but I feel temporarily challenging non-API usage could have at least helped without impacting users too much?
Re: Understanding the recent DDoS attack against Read the Docs
#6I'm curious if anybody could speculate who would be attacking a documentation silo, and to what end?
I'm the author of the blog. I don't know. Internally, we were half joking that we were going to get ransom notice, but we never did. The only thing that sort of correlates with this attack is that before it started, we began rolling out some slightly more aggressive rate limits one by one. This was mostly because anytime any new "company" thinks they're going to catchup with Claude/OpenAI, they scrape us very aggress…
Just curious, if you're tolerant of scraping, do you make an archive of all your content available so that scraping is unnecessary, and if so do the scrapers prefer that?
Re: Understanding the recent DDoS attack against Read the Docs
#7I'm curious if anybody could speculate who would be attacking a documentation silo, and to what end?
I'm the author of the blog. I don't know. Internally, we were half joking that we were going to get ransom notice, but we never did. The only thing that sort of correlates with this attack is that before it started, we began rolling out some slightly more aggressive rate limits one by one. This was mostly because anytime any new "company" thinks they're going to catchup with Claude/OpenAI, they scrape us very aggress…
Good to know. I use your site (with a manual transmission user-agent) often, and it's fantastic. Thanks for your work and the writeup!
Re: Understanding the recent DDoS attack against Read the Docs
#8Earlier quoted context omitted.
I'm the author of the blog. I don't know. Internally, we were half joking that we were going to get ransom notice, but we never did. The only thing that sort of correlates with this attack is that before it started, we began rolling out some slightly more aggressive rate limits one by one. This was mostly because anytime any new "company" thinks they're going to catchup with Claude/OpenAI, they scrape us very aggress…
Just curious, if you're tolerant of scraping, do you make an archive of all your content available so that scraping is unnecessary, and if so do the scrapers prefer that?
It's terabytes of content and other than we're the host not really related to each other. However, for most projects, it's possible to download a zip file of all the HTML docs for that project. We have a lower rate limit to pull these, but a scraper can pull thousands of docs at once. We only host a few hundred thousand projects so pulling a zip of the latest docs for all of them could be done in a day or two at a very reasonable rate.
It's also possible to request the docs already processed into markdown[1]. Lastly, basically all of the docs come from Git. A smart scraper could just clone a project's repo.
[1] https://docs.readthedocs.com/platform/stable/reference/markd...
Re: Understanding the recent DDoS attack against Read the Docs
#9Re: Understanding the recent DDoS attack against Read the Docs
#10Interesting that the Under Attack Mode wasn’t used at all here. I understand not wanting to break APIs but I feel temporarily challenging non-API usage could have at least helped without impacting users too much?
I talked about that directly in the post. We didn't want to just challenge everyone. We use JS challenges but we try to use them sparingly. The rest of the ops team and I were fighting to stay up but it never got so bad that it was a choice between complete outage and using the Under Attack mode.