Has anyone/any company asked you to buy their anti-DdoS service recent days/weeks?
Let us serve you, but don't bring us down
41–50 of 255 posts
Re: Let us serve you, but don't bring us down
#42What is the legal ground for archive.org to copy websites? Shouldn't copyright forbid that? They don't even respect robots.txt. So content creators can't even opt out of that. Not that copyright would have copyright holders having to opt out of copying in the first place. How have they not been sued out of existance yet?
[dead]
Re: Let us serve you, but don't bring us down
#43Unpopular opinion: Severely rate limit retrieving the files from the website / HTTP endpoint, and loudly point towards downloading the files via torrents. The torrent protocol was meant to relieve this level of server load in mind.
and loudly point towards downloading the files via torrents. By "loudly", perhaps actually redirect to the .torrent instead, for those who trip the rate limit?
...that can work.
Re: Let us serve you, but don't bring us down
#44I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, b…
Temporarily add 500ms of latency to all ipv6 users, backoff timers for ipv4 addresses. Since there's only 4 billion v4 addresses, it's easy enough to just track them all in a sqlite db.
Mostly works, but there do exist a couple of botnets that contain 1 million compromised machines. If each makes one request before hitting backoff, spread evenly throughout the day, that's about 10 QPS alone before they use an IP number twice. But they tend to not actually level out their usage (which is a bummer - if they did they could have kept using it). Instead they hit with a lot of parallel queries all at once.
There's only so much you can really do when your underlying resource is so limited. Luckily the value of the query is lower than the cost of a recaptcha solve, so the attackers moved on to some other target.
Ironically I could now turn off the endpoint protection (or have it responsive to traffic load), until the attackers return. I shall not go into too many details, no need to give people a map.
Re: Let us serve you, but don't bring us down
#45I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, b…
Captcha seems like overkill. Were you not able to implement a JS fingerprinting / bot check before captcha.
Yeah, "recaptcha" in this case is shorthand for a lot of stuff we did to harden the endpoint that ultimately represents a minor shift in balance from no friction to some friction for otherwise legitimate users, but a pretty significant drop in illegitimate traffic.
The main idea here is that at some point just leaving the scrape running in a way that didn't overwhelm my backend would have resulted in me not caring enough to do that. But now they get nothing. Even if you're borrowing bandwidth and not paying, you should be a good neighbor is all.
Re: Let us serve you, but don't bring us down
#46Unpopular opinion: Severely rate limit retrieving the files from the website / HTTP endpoint, and loudly point towards downloading the files via torrents. The torrent protocol was meant to relieve this level of server load in mind.
[flagged]
Re: Let us serve you, but don't bring us down
#47I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, b…
Maybe mCaptcha [0] is worth a look. It applies a Proof-of-Work like algorithm (not blockchain-related) which makes it very expensive for scrapers to get data in bulk, but poses least amount of friction to individual users. The project is implemented in Rust and received NGI.eu/NLnet funding. I don't know its state of production-readiness, but Codeberg.org is considering using it (this choice is informed by higher res…
Really interesting! How efficiently does a web browser compute the PoW? I'm concerned that a bot would use an efficient GPU implementation while real users would run an inefficient JS/webcrypto version.
Re: Let us serve you, but don't bring us down
#48OCRs of PD material? Is someone about to train a new AI model, perhaps? I'd think the Archive would be happy to arrange for receiving hard drives, filling them up, and sending them back. They seem like a very helpful bunch.
Re: Let us serve you, but don't bring us down
#49I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, b…
Maybe mCaptcha [0] is worth a look. It applies a Proof-of-Work like algorithm (not blockchain-related) which makes it very expensive for scrapers to get data in bulk, but poses least amount of friction to individual users. The project is implemented in Rust and received NGI.eu/NLnet funding. I don't know its state of production-readiness, but Codeberg.org is considering using it (this choice is informed by higher res…
If it doesn’t involve my free labor training machine learning models like reCAPTCHA and hCAPTCHA and I can still visit sites from a non-Western IP, then it’s already an improvement.
Re: Let us serve you, but don't bring us down
#50I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, b…
Maybe mCaptcha [0] is worth a look. It applies a Proof-of-Work like algorithm (not blockchain-related) which makes it very expensive for scrapers to get data in bulk, but poses least amount of friction to individual users. The project is implemented in Rust and received NGI.eu/NLnet funding. I don't know its state of production-readiness, but Codeberg.org is considering using it (this choice is informed by higher res…
how does it work? perhaps it's proprietary that the authors wouldn't want to disclose, but I did not see anything referenced in the site. I wouldn't plug it into mine without knowing g