131–140 of 255 posts
I recall tripping the rate limiter at least once by accident - I was clicking around the archived site for reference while working on creating a restored version.
Unpopular opinion: Severely rate limit retrieving the files from the website / HTTP endpoint, and loudly point towards downloading the files via torrents. The torrent protocol was meant to relieve this level of server load in mind.
Torrents have the habit of disappearing when no users keep them alive. It happened to me enough times to be wary of such solution. If there's a way to keep them alive regardless of interest I'm all for it.
Earlier quoted context omitted.
Indeed. Tens of thousands of requests per second from just 64 hosts? So they allow individual hosts to make hundreds of requests per second, sustained? That sounds crazy. Even for a burst limit hundreds per second would be extremely high.
Archive.org is a core utility for the web to the point where Wikipedia and many other sites would collapse without it in the sense that many if not most of their outbound links would be dead forever. I’m pretty sure it would even impact the US justice system [1]. Obviously judges aren’t going to have to worry about reasonable rate limits but if these DDoSes are rare, I’d much rather they dealt with them on a case by…
What an old fashioned concern! Don't you know that the US justice system uses ChatGPT as an archive retrieval system nowadays? /s
Earlier quoted context omitted.
In my opinion, PoW is the only reliable way, to avoid DDoS attacks. Scraping too much, too quickly is a light form of DDoS, although not intentional. PoW was invented in 2006 exactly for that purpose. The genius of bitcoin (not BTC) is that it provides an organized and practical way, for PoW to be used by everyone on the planet. Some people find it strange, because there is an imaginary token created out of pure noth…
I hate to be a pedant but I see it used wrong a lot. "DDOS" stands for Distributed denial of service, specifically indicating the traffic is coming from many sources (wide range of ip), which is what makes it so hard to defend against. Someone scraping too fast would be performing an unintentional DOS, because they probably arent scraping using a botnet (and if they were they probably do in fact intend to attack)
Earlier quoted context omitted.
Requiring every user to compute it's own PoW is a terrible idea. It defeats the whole purpose. One's person expensive computation is another person's almost free computation. I commented in the past about it: "At first glance, yes, we can create intentionally expensive computations without relying on a blockchain, that would serve the same purpose. In reality we cannot. Special computer hardware (ASICs) could generat…
Specialized ASICs are not easy to come up with, what's more if it's in a tug of war you can expect those to suddenly become obsolete, rendering this approach too costly for a scrapper. Also you may have missed the part in the original link where they were using AWS to do the scrapping, so no ASICs there, and you pay by the minute, two things that would make PoW a valid countermeasure, no "free computation" in that se…
Earlier quoted context omitted.
> This coin is not listed on Binance for trade and service. An obscure coin my users cannot purchase through major exchanges is a really poor solution. Sorry, that makes no sense.
I answered on the other comment, but at that point, in case there is a blockchain, which supports a millionth of a cent transaction, then exchanges (Binance, Coinbase etc) are not so useful. If every person just needs one cent for a million http requests, then one guy in your neighbourhood or your town, or your city might have some of it, you message him and he will send you a cent for free. You buy him a coffee, and…
I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, b…
Maybe mCaptcha [0] is worth a look. It applies a Proof-of-Work like algorithm (not blockchain-related) which makes it very expensive for scrapers to get data in bulk, but poses least amount of friction to individual users. The project is implemented in Rust and received NGI.eu/NLnet funding. I don't know its state of production-readiness, but Codeberg.org is considering using it (this choice is informed by higher res…
Earlier quoted context omitted.
I hate to be a pedant but I see it used wrong a lot. "DDOS" stands for Distributed denial of service, specifically indicating the traffic is coming from many sources (wide range of ip), which is what makes it so hard to defend against. Someone scraping too fast would be performing an unintentional DOS, because they probably arent scraping using a botnet (and if they were they probably do in fact intend to attack)
And if they’re scraping using “serverless” compute? It’s distributed, but could be used without “meaning” to attack. That seems to be what happened to Archive (they said AWS, could be ECS or Lambda, idk). FWIW this thread inspired me to implement Cloudflare Turnstile on one of my pages - highly recommend. As compared to reCAPTCHA your users are never wasting their life away clicking on traffic signs, and as compared…
Unpopular opinion: Severely rate limit retrieving the files from the website / HTTP endpoint, and loudly point towards downloading the files via torrents. The torrent protocol was meant to relieve this level of server load in mind.
Torrents have the habit of disappearing when no users keep them alive. It happened to me enough times to be wary of such solution. If there's a way to keep them alive regardless of interest I'm all for it.
This approach allows the internet archive to effectively rate limit hosts without making content unavailable entirely, and allows others to help carry the load.