Sounds like they don't have rate limiting implemented? That seems odd, and it's also surprising it isn't talked about in the post.
Precisely. What kinds of legitimate processes would require this kind of utility from Archive.org?
Let us serve you, but don't bring us down
51–60 of 255 posts
Re: Let us serve you, but don't bring us down
#52Sounds like they don't have rate limiting implemented? That seems odd, and it's also surprising it isn't talked about in the post.
[deleted]
Re: Let us serve you, but don't bring us down
#53OCRs of PD material? Is someone about to train a new AI model, perhaps? I'd think the Archive would be happy to arrange for receiving hard drives, filling them up, and sending them back. They seem like a very helpful bunch.
It's ChatGPT itself....learning....evolving...:)
Re: Let us serve you, but don't bring us down
#54Earlier quoted context omitted.
Maybe mCaptcha [0] is worth a look. It applies a Proof-of-Work like algorithm (not blockchain-related) which makes it very expensive for scrapers to get data in bulk, but poses least amount of friction to individual users. The project is implemented in Rust and received NGI.eu/NLnet funding. I don't know its state of production-readiness, but Codeberg.org is considering using it (this choice is informed by higher res…
how does it work? perhaps it's proprietary that the authors wouldn't want to disclose, but I did not see anything referenced in the site. I wouldn't plug it into mine without knowing g
It isn't proprietary [0]. There is a matrix channel [1] for the project to ask questions.
Re: Let us serve you, but don't bring us down
#55Earlier quoted context omitted.
Indeed. Tens of thousands of requests per second from just 64 hosts? So they allow individual hosts to make hundreds of requests per second, sustained? That sounds crazy. Even for a burst limit hundreds per second would be extremely high.
Archive.org is a core utility for the web to the point where Wikipedia and many other sites would collapse without it in the sense that many if not most of their outbound links would be dead forever. I’m pretty sure it would even impact the US justice system [1]. Obviously judges aren’t going to have to worry about reasonable rate limits but if these DDoSes are rare, I’d much rather they dealt with them on a case by…
Rate limiting is (usually) done on a per-host basis. The only users who would be adversely affected are those who perform hundreds of requests per second from a single system.
Virtually every major website has per-host inbound request limits. This is completely standard practice and Archive.org is the odd one here.
And DDoS isn't the only concern. Legitimate users that run poorly written Python scripts that make insane numbers of requests can hog server resources, and rate limits with appropriate error messages linking to resources documenting efficient access patterns can improve the experience for everyone, and drastically cut costs for the service operators.
Re: Let us serve you, but don't bring us down
#56Earlier quoted context omitted.
Maybe mCaptcha [0] is worth a look. It applies a Proof-of-Work like algorithm (not blockchain-related) which makes it very expensive for scrapers to get data in bulk, but poses least amount of friction to individual users. The project is implemented in Rust and received NGI.eu/NLnet funding. I don't know its state of production-readiness, but Codeberg.org is considering using it (this choice is informed by higher res…
Really interesting! How efficiently does a web browser compute the PoW? I'm concerned that a bot would use an efficient GPU implementation while real users would run an inefficient JS/webcrypto version.
I tried to implement PoW in browser for the same concept. I think it's probably at least marginally useful, but practically you're limited by WebCrypto I think, which outperformed anything I could find in pure JS or WASM. The disadvantage of WebCrypto is that there's a limited set of algorithms you can use and also, understandably, calculating hashes is async, so if you want a lot of rounds, you'll spend a lot of time jumping in and out of the event loop. It's still probably a useful speed bump or price increase for expensive operations: what might cost a few seconds on a typical phone or desktop is probably at least enough to act as a speed bump for attackers to perform it quickly (especially combined with other measures like throttling and progressively increasing PoW difficulty for an IP.) Maybe WebGPU could change this, but I'm weary of relying on all users having a fast GPU enabled in their browser just for this use case.
Though thinking about it, I wonder if there is a hybrid: start with a difficulty that's just a few seconds for CPU/WebCrypto and ramp up quickly, but also support WebGPU where possible so that web users on abusive connections may still succeed? I am not sure though, I guess this depends on the feasibility of using WebGPU and etc.
Re: Let us serve you, but don't bring us down
#57I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, b…
I had similar issue for one of our clients.
My strategy to not affect legit users was to enable mitigations if global traffic crossed some threshold.
eg. in your case this could mean if traffic is above eg. 75QPS then captcha is enabled, and if it's below that it's disabled.
I don't know what tech stack you are using, but nice trick that i figured out was to abuse rate limiting to detect global traffic (doing if branch with rate limit with const as client id)
Re: Let us serve you, but don't bring us down
#58Unpopular opinion: Severely rate limit retrieving the files from the website / HTTP endpoint, and loudly point towards downloading the files via torrents. The torrent protocol was meant to relieve this level of server load in mind.
[flagged]
Torrents aren't illegal inherently; the content downloaded are often copyright violations, yes, but using torrenting protocol itself isn't illegal. Most Linux distributions provide a torrent option to download their OS image file.
Re: Let us serve you, but don't bring us down
#59[deleted]
Re: Let us serve you, but don't bring us down
#60I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, b…
Maybe mCaptcha [0] is worth a look. It applies a Proof-of-Work like algorithm (not blockchain-related) which makes it very expensive for scrapers to get data in bulk, but poses least amount of friction to individual users. The project is implemented in Rust and received NGI.eu/NLnet funding. I don't know its state of production-readiness, but Codeberg.org is considering using it (this choice is informed by higher res…
In my opinion, PoW is the only reliable way, to avoid DDoS attacks. Scraping too much, too quickly is a light form of DDoS, although not intentional. PoW was invented in 2006 exactly for that purpose.
The genius of bitcoin (not BTC) is that it provides an organized and practical way, for PoW to be used by everyone on the planet. Some people find it strange, because there is an imaginary token created out of pure nothing which represents the PoW. It would work fine, without that imaginary token i.e. bitcoin, just by dollars of euros, but it wouldn't be internet native. The point is always to just send a minimal PoW with every http or tcp/ip request.