151–160 of 255 posts
I'm not an expert in networking, but isn't this scenario Cloudflare's bread and butter? Is there any reason to not outsource this problem?
10,000 requests/seconds is nowadays an average number for load testing in the companies I work with, for websites / mobile apps deployed at this scale
I have a side project that scrapes thousands and thousands of pages of a single website. So as not to piss them off (and so they don't try to block me), my script will take about 6 hours. Between each page fetch it sleeps for a small, random amount of time. It's been working like that for years.
Wish more people did like you!
Just yesterday there was a comment here on HN [1] about https://jsonip.com , which is essentially supported by a single person (all operational costs included) and gets abused in a somewhat similar manner. I am not even sure what to think: do the folks not understand what they do, or are they just bluntly ignorant of it? [1] https://news.ycombinator.com/item?id=36092417
> o the folks not understand what they do, or are they just bluntly ignorant of it? They don't care. And yes I've heard stories of people "finding a service" that does something basic and just dumping their whole traffic onto it (of course they play the victim once they're found out)
The fact that IA is so liberal with their scraping policies is laudable, because on the other end of the spectrum lies Wikimedia, where you're lucky if you manage to download their dumps at 500kbps, which makes their dumps pretty much impossible to obtain...
over 50MB in 1minute (6MB/hr).
http://dumps.wikimedia.your.org/enwiki/20220820/
They also seed torrents
I have a side project that scrapes thousands and thousands of pages of a single website. So as not to piss them off (and so they don't try to block me), my script will take about 6 hours. Between each page fetch it sleeps for a small, random amount of time. It's been working like that for years.
Earlier quoted context omitted.
In my opinion, PoW is the only reliable way, to avoid DDoS attacks. Scraping too much, too quickly is a light form of DDoS, although not intentional. PoW was invented in 2006 exactly for that purpose. The genius of bitcoin (not BTC) is that it provides an organized and practical way, for PoW to be used by everyone on the planet. Some people find it strange, because there is an imaginary token created out of pure noth…
Proof-of-work was originally introduced by Adam Back's hashcash, to fight email spam. Bitcoin is an evolution of that idea applied to digital cash. https://en.wikipedia.org//wiki/Hashcash
Just yesterday there was a comment here on HN [1] about https://jsonip.com , which is essentially supported by a single person (all operational costs included) and gets abused in a somewhat similar manner. I am not even sure what to think: do the folks not understand what they do, or are they just bluntly ignorant of it? [1] https://news.ycombinator.com/item?id=36092417
I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, b…
If you can detect it reliably, poisoning them with bad answers might be enough to cause them to find an easier target.
He knew that was the reason for the intentional DDoS due to messages (along the lines of “think it is funny to poison my information do you?”) in the query string of the bulk requests. Like unpleasant fools making a big noise in shops because they aren't served immediately after cutting in line or some such, the entitled can be quite petty when actively taken to task for the inconvenience they cause.
Passive defences are safer in that regard, assuming you don't get into an arms race with the scrapers, though are unfortunately more likely to mildly inconvenience your good users.
Earlier quoted context omitted.
If you can detect it reliably, poisoning them with bad answers might be enough to cause them to find an easier target.
Be careful with that. A friend once did something similar to an active scraper who then went off and set his site up for a full blown DDoS. He knew that was the reason for the intentional DDoS due to messages (along the lines of “think it is funny to poison my information do you?”) in the query string of the bulk requests. Like unpleasant fools making a big noise in shops because they aren't served immediately after…
Though I did have success once stopping hot-linked images by serving up images that didn't fit well with the sensibilities of the internet forum that was hotlinking them. It had the advantage of looking like the people in the forum posting had deliberately chosen to post the image. Serving different images depending on client ip made for fun too, as they argued with each other about why they posted "that".
The fact that IA is so liberal with their scraping policies is laudable, because on the other end of the spectrum lies Wikimedia, where you're lucky if you manage to download their dumps at 500kbps, which makes their dumps pretty much impossible to obtain...
$ curl https://dumps.wikimedia.org/enwiki/20230520/enwiki-20230520-pages-articles-multistream.xml.bz2 -o/dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
2 20.4G 2 476M 0 0 4387k 0 1:21:38 0:01:51 1:19:47 4393k^C
$ curl https://wikidata.aerotechnet.com/enwiki/20230520/enwiki-20230520-pages-articles-multistream.xml.bz2 -o/dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 20.4G 0 45.9M 0 0 1953k 0 3:03:23 0:00:24 3:02:59 2257k^C
$ curl https://mirror.clarkson.edu/wikimedia/enwiki/20230520/enwiki-20230520-pages-articles-multistream.xml.bz2 -o/dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
1 20.4G 1 344M 0 0 35.2M 0 0:09:56 0:00:09 0:09:47 36.8M^C