221–230 of 255 posts
Unpopular opinion: Severely rate limit retrieving the files from the website / HTTP endpoint, and loudly point towards downloading the files via torrents. The torrent protocol was meant to relieve this level of server load in mind.
and loudly point towards downloading the files via torrents. By "loudly", perhaps actually redirect to the .torrent instead, for those who trip the rate limit?
Earlier quoted context omitted.
Torrents have the habit of disappearing when no users keep them alive. It happened to me enough times to be wary of such solution. If there's a way to keep them alive regardless of interest I'm all for it.
This is what "proof of storage" or "proof of data availability" blockchain networks are for. They use economic incentives to continuously pay nodes a small amount to store some data and keep it available, and the cryptographic sampling mechanism ensures that less popular data must remain in the available dataset for nodes to be paid, even if it is rarely requested in full.
Is there an open-source rate limiter that works well for sites large and small? It just strikes me as surprising that sites are still dealing with problems like this in 2023. The idea that a site is still manually having to identify a set of IP addresses and block them with human intervention seems absolutely archaic by this point. And I don't think different sites have particularly different needs here... basic patt…
I think rate limiting is the wrong approach. If there’s too much load you need to manage the queue
I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, b…
I think you could make your requests two step. If somebody wants to access endpoint you might send him a challenge first. Random text. The client must append to the text some other text chosen by him, so that when you calculate sha256 on concatenated text, first byte or two of it will be zeros. To access your actual endpoint client needs to send that generated text and you can check it if it results in the required n…
Earlier quoted context omitted.
I think you could make your requests two step. If somebody wants to access endpoint you might send him a challenge first. Random text. The client must append to the text some other text chosen by him, so that when you calculate sha256 on concatenated text, first byte or two of it will be zeros. To access your actual endpoint client needs to send that generated text and you can check it if it results in the required n…
Because it doesn’t accomplish anything. Things take longer for honest users while botnet abusers don’t even notice that the rented hardware is burning more CPU. Nor does it matter because each request still goes through.
Generating challenges is really cheap. Calculating a single SHA256 or sth out of a querystring+salt (or better yet 128bit SipHash). Generation and validation can be done on separate layer/server so requests without valid PoW won't even register on your main system.
How could they not have rate limiting?
Earlier quoted context omitted.
Some zealous systems will infer a very regular request rate as coming from automated services and block them, no matter how gentle the rate.
Is this speculation, or something that has actually been seen?
Funny story - at work we once had a huge spike in requests from a single IP. We all crowded around, thinking it was some malicious hacker from France. How exciting - we're now interesting enough to warrant a DoS! Turns out another team in the company was just pulling all our data into Algolia to improve search. They were clearly not very courteous!
So on the other end (building APIs) I certainly do pay attention to traffic and have Grafana alerts set up around it.
I have a side project that scrapes thousands and thousands of pages of a single website. So as not to piss them off (and so they don't try to block me), my script will take about 6 hours. Between each page fetch it sleeps for a small, random amount of time. It's been working like that for years.
I am sorry, admins! I won't do it next time.
This is a job for HTTP 429!
Or 420 :P
I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, b…
Maybe mCaptcha [0] is worth a look. It applies a Proof-of-Work like algorithm (not blockchain-related) which makes it very expensive for scrapers to get data in bulk, but poses least amount of friction to individual users. The project is implemented in Rust and received NGI.eu/NLnet funding. I don't know its state of production-readiness, but Codeberg.org is considering using it (this choice is informed by higher res…
If you're going to waste my energy to do proof-of-work anyway, I'd rather you use it for something useful (even mining crypto-currency to pay for server costs) rather than let it go to waste.