171–180 of 255 posts
I have a side project that scrapes thousands and thousands of pages of a single website. So as not to piss them off (and so they don't try to block me), my script will take about 6 hours. Between each page fetch it sleeps for a small, random amount of time. It's been working like that for years.
Just yesterday there was a comment here on HN [1] about https://jsonip.com , which is essentially supported by a single person (all operational costs included) and gets abused in a somewhat similar manner. I am not even sure what to think: do the folks not understand what they do, or are they just bluntly ignorant of it? [1] https://news.ycombinator.com/item?id=36092417
Yeah I'm getting this shit too with Marginalia Search. I'm getting about 2-2.5M queries per day that are definitely from bots that would 100% sink my server if they somehow go through the bot mitigation. It peaks at hundreds of search queries per second. To be clear these are search queries, and search queries typically trigger disk reads of about ~10-20 Mb. I get about 20,000 queries per day that may be human.
I just wonder, is archive.org getting any government grant money? If they aren't they should. And I'm not even talking about just the US. All sorts of countries (US, UK, Germany - to name the few) and international organisations like EU pour hundreds of millions into "cultural projects" of very questionable value. How about they actually fund something really worthy of preservation? Of course it is archive.org role t…
Unpopular opinion: Severely rate limit retrieving the files from the website / HTTP endpoint, and loudly point towards downloading the files via torrents. The torrent protocol was meant to relieve this level of server load in mind.
Depending on the type of content you download, the chances of anyone else already seeding that might actually be zero.
I just wonder, is archive.org getting any government grant money? If they aren't they should. And I'm not even talking about just the US. All sorts of countries (US, UK, Germany - to name the few) and international organisations like EU pour hundreds of millions into "cultural projects" of very questionable value. How about they actually fund something really worthy of preservation? Of course it is archive.org role t…
1: https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
Earlier quoted context omitted.
Proof-of-work was originally introduced by Adam Back's hashcash, to fight email spam. Bitcoin is an evolution of that idea applied to digital cash. https://en.wikipedia.org//wiki/Hashcash
Would it be possible to use bitcoin pow as a replacement for captcha? It both blocks ddos and create an income through mining (client side) for the host.
Earlier quoted context omitted.
In my opinion, PoW is the only reliable way, to avoid DDoS attacks. Scraping too much, too quickly is a light form of DDoS, although not intentional. PoW was invented in 2006 exactly for that purpose. The genius of bitcoin (not BTC) is that it provides an organized and practical way, for PoW to be used by everyone on the planet. Some people find it strange, because there is an imaginary token created out of pure noth…
Proof-of-work was originally introduced by Adam Back's hashcash, to fight email spam. Bitcoin is an evolution of that idea applied to digital cash. https://en.wikipedia.org//wiki/Hashcash
Earlier quoted context omitted.
Maybe mCaptcha [0] is worth a look. It applies a Proof-of-Work like algorithm (not blockchain-related) which makes it very expensive for scrapers to get data in bulk, but poses least amount of friction to individual users. The project is implemented in Rust and received NGI.eu/NLnet funding. I don't know its state of production-readiness, but Codeberg.org is considering using it (this choice is informed by higher res…
I got interested in mCaptcha, followed your link, but couldn’t find anywhere an example of what the end user would deal with. What kind of PoW are we talking about?
Earlier quoted context omitted.
Proof-of-work was originally introduced by Adam Back's hashcash, to fight email spam. Bitcoin is an evolution of that idea applied to digital cash. https://en.wikipedia.org//wiki/Hashcash
Would it be possible to use bitcoin pow as a replacement for captcha? It both blocks ddos and create an income through mining (client side) for the host.
Hardware specialization breaks the economics behind a CAPTCHA. To fight that you need to use a PoW that hasn't been ASIC'd yet, and be willing to change PoW functions at the drop of a hat. PoW functions that stress memory or cache are also helpful here, though you run the risk of browsers flagging you as a cryptominer (which is technically correct, even if economically wrong).
There's no reason for Archive.org to allow any traffic from AWS, Azure or Google Cloud. Archive.org should just block the CIDRs for all those networks. Most traffic that comes from them is scraping traffic and usually malicious.