Earlier quoted context omitted.
Details?
found this: https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
Seems reasonable
Let us serve you, but don't bring us down
11–20 of 255 posts
Re: Let us serve you, but don't bring us down
#12Pretty sure it was an attack, not legitimate bulk use of the website.
Re: Let us serve you, but don't bring us down
#13I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, b…
Captcha seems like overkill. Were you not able to implement a JS fingerprinting / bot check before captcha.
Re: Let us serve you, but don't bring us down
#14I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, b…
Temporarily add 500ms of latency to all ipv6 users, backoff timers for ipv4 addresses. Since there's only 4 billion v4 addresses, it's easy enough to just track them all in a sqlite db.
I don't think that works if your attacker has millions of IPs and is only using 200 per second.
Re: Let us serve you, but don't bring us down
#15I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, b…
did you ever look inside those queries? were they the same, just repeated ad nauseam just polling in case their status had changed?
Re: Let us serve you, but don't bring us down
#16I'm not an expert in networking, but isn't this scenario Cloudflare's bread and butter? Is there any reason to not outsource this problem?
Re: Let us serve you, but don't bring us down
#17Earlier quoted context omitted.
Temporarily add 500ms of latency to all ipv6 users, backoff timers for ipv4 addresses. Since there's only 4 billion v4 addresses, it's easy enough to just track them all in a sqlite db.
I don't think that works if your attacker has millions of IPs and is only using 200 per second.
How are they getting millions of ipv4 addresses? IIUC that’s at least the equivalent of a /12 block. Do those shady residential proxies really operate at that scale?
If they’re ipv6 address wouldn’t they be safe to block across large ranges?
Re: Let us serve you, but don't bring us down
#18What is the legal ground for archive.org to copy websites? Shouldn't copyright forbid that? They don't even respect robots.txt. So content creators can't even opt out of that. Not that copyright would have copyright holders having to opt out of copying in the first place. How have they not been sued out of existance yet?
Same way a library does. The information is made available to anyone who wants it, the copyrights are maintained intact, and they don’t attempt to profit from the material.
Copyright law specifically allows for libraries and archives to make copies of copyrighted material.
Without such laws, without libraries, knowledge could not be guaranteed to be shared freely among the public, resulting in ever growing knowledge and education gaps between those with means and those without.
Edit: Since you asked for the legal ground, here it is specifically:
https://www.law.cornell.edu/uscode/text/17/108
And here’s is further discussion by the copyright office itself:
https://www.copyright.gov/policy/section108/discussion-docum...
Congress and the copyright office have made it an important part of copyright law to protect archival, library and fair use doctrine.
Re: Let us serve you, but don't bring us down
#19[flagged]
Re: Let us serve you, but don't bring us down
#20Huh, meanwhile, I recently wanted to grab ~200 pages from the archive where I put a 1-2 minute delay between the requests and had a bunch of repeated failures.