191–200 of 255 posts
There's no reason for Archive.org to allow any traffic from AWS, Azure or Google Cloud. Archive.org should just block the CIDRs for all those networks. Most traffic that comes from them is scraping traffic and usually malicious.
Earlier quoted context omitted.
There are various underfunded digital archives in the EU and elsewhere that have legal duties to preserve online content relating to their specific countries. They may be "of very questionable value" to you but your solution to remove funding from them to channel it to a much wealthier organisation in a wealthier country is neither ethical, legal or practical.
This is almost completely unresponsive to what parent actually wrote.
The fact that IA is so liberal with their scraping policies is laudable, because on the other end of the spectrum lies Wikimedia, where you're lucky if you manage to download their dumps at 500kbps, which makes their dumps pretty much impossible to obtain...
I just tried it. over 50MB in 1minute (6MB/hr). http://dumps.wikimedia.your.org/enwiki/20220820/ They also seed torrents
I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, b…
If somebody wants to access endpoint you might send him a challenge first. Random text. The client must append to the text some other text chosen by him, so that when you calculate sha256 on concatenated text, first byte or two of it will be zeros.
To access your actual endpoint client needs to send that generated text and you can check it if it results in the required number of zeros. You might demand more zeros in times of heavier load. Each additional bit that you require to be zero increases number of random attempts to find the text by a factor of two.
To make stuff easier for yourself the challenge text instead of being random might be a hash of clients request parameters and some secret salt. Then you don't have to remember the text or any context at all between client requests. You can regenerate it when the client sends second requests with answer.
Honestly I don't know why this isn't a standard option in frameworks for building public facing apis that are expected to be used reasonably.
I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, b…
It just strikes me as surprising that sites are still dealing with problems like this in 2023.
The idea that a site is still manually having to identify a set of IP addresses and block them with human intervention seems absolutely archaic by this point.
And I don't think different sites have particularly different needs here... basic pattern-matching heuristics can identify IP addresses/blocks (plus things like HTTP headers) that are suddenly ramping up requests, and use CAPTCHAs to allow legitimate users through when one IP address is doing NAT for many users behind it. Really the main choice is just whether to block spidering as much as possible, always allow it as long as it's at a reasonable speed, or limit it to a standard whitelist of known orgs (Google, Bing, Facebook, Internet Archive, etc.).
It just strikes me as odd that when you follow a basic tutorial for installing Apache on Linux on Digital Ocean, rate limiting isn't part of it. It seems like it should be almost as basic as an HTTPS certificate by this point.
I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, b…
There's a demand, why not supply it, and make money while you are at it? This reminds me of the RMT driven botting problem in WoW (World of Warcraft). Instead of fighting the neverending game of cat and mouse against botters, Blizzard just decided to supply the long reprimanded demand for in-game currency by creating the WoW token, and they make money while they're at.
Earlier quoted context omitted.
I think he or she rather meant some other cultural projects than digital archives. There are indeed lots of cultural projects that get funding, that I consider not that important in comparison, but of course those people involved would think different. (Opera for example is heavily subsidized)
Opera the theater or Opera the web browser?
For real? I am not against subsidicing art and culture, but I am against selective subsidicing. For example in germany there is a strong divide into "serious art" like opera and classical music that gets lots of money direct or indirectly - and trivial art, everyone else. Getting allmost nothing. So it boils down to taste and the favourite culture of the establishment. But there is so much other good music and performers besides the mainstream out there, who gets categorized into "entertainment" and have to struggle on their own.
So back on topic, I would be fine with taking money from opera to give it to internet archives. But of course, I rather would have more money for everyone involved in arts and culture.
Earlier quoted context omitted.
I got interested in mCaptcha, followed your link, but couldn’t find anywhere an example of what the end user would deal with. What kind of PoW are we talking about?
You can find some demos linked on github [1], the simplest being: https://demo.mcaptcha.org/widget/?sitekey=pHy0AktWyOKuxZDzFf... [1] https://github.com/mCaptcha/mCaptcha#client-side-widget
1. whatever interface you expose to the public, some will abuse it
2. rules/laws/countermeasures are then added and enforced, often universally (though sometimes inconsistently)
3. which then bites, annoys, insults or otherise burdens or adds to the prices paid by ALL the OTHER non-abusing users pf that same interface
always. eventually. every time