Let us serve you, but don't bring us down
blog.archive.org
Let us serve you, but don't bring us down
1–10 of 255 posts
Re: Let us serve you, but don't bring us down
#2Sounds like they don't have rate limiting implemented? That seems odd, and it's also surprising it isn't talked about in the post.
Re: Let us serve you, but don't bring us down
#3What is the legal ground for archive.org to copy websites? Shouldn't copyright forbid that?
They don't even respect robots.txt. So content creators can't even opt out of that. Not that copyright would have copyright holders having to opt out of copying in the first place.
How have they not been sued out of existance yet?
Re: Let us serve you, but don't bring us down
#4What is the legal ground for archive.org to copy websites? Shouldn't copyright forbid that? They don't even respect robots.txt. So content creators can't even opt out of that. Not that copyright would have copyright holders having to opt out of copying in the first place. How have they not been sued out of existance yet?
Details?
Re: Let us serve you, but don't bring us down
#5I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1.
I was keeping an eye on it, because we are hard-capped at 100 QPS to our provider, beyond that and they start dropping our traffic (and it is an outside provider, bundling domain registries like verisign and stuff), which makes regular users break if their traffic gets unlucky.
Anyway, after a week of 40qps, they start spiking to 200+, and we pull the plug on the whole thing: now each request to our endpoint requires a recaptcha token. This is not great (more friction for legit users = more churn) but it is successful. IF they had only kept their QPS low, nobody would have cared. I wanted to send some kind of response code like, "nearing quota".
FTR before people ask: it was quite difficult to stop this particular attack, since it worked like a DDOS, smeared across a _large_ number of ipv4 and ipv6 requesters. 50 QPS just isn't enough quota to do stuff like reactively banning IP numbers if the attacker has millions of IPs available.
Re: Let us serve you, but don't bring us down
#6What is the legal ground for archive.org to copy websites? Shouldn't copyright forbid that? They don't even respect robots.txt. So content creators can't even opt out of that. Not that copyright would have copyright holders having to opt out of copying in the first place. How have they not been sued out of existance yet?
Details?
Re: Let us serve you, but don't bring us down
#7I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, b…
QPS seems to be "queries per second".
I first thought it was something like "quadrillion bytes per second" or some newer over-the-top data measurement :)
Re: Let us serve you, but don't bring us down
#8Sounds like they don't have rate limiting implemented? That seems odd, and it's also surprising it isn't talked about in the post.
Indeed. Tens of thousands of requests per second from just 64 hosts? So they allow individual hosts to make hundreds of requests per second, sustained? That sounds crazy. Even for a burst limit hundreds per second would be extremely high.
Re: Let us serve you, but don't bring us down
#9I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, b…
Temporarily add 500ms of latency to all ipv6 users, backoff timers for ipv4 addresses. Since there's only 4 billion v4 addresses, it's easy enough to just track them all in a sqlite db.
Re: Let us serve you, but don't bring us down
#10> "Error establishing database connection"
Well, that's ironic.
... Anyone have an archive.org link of the page?