> "Error establishing database connection" Well, that's ironic. ... Anyone have an archive.org link of the page?
[deleted]
Let us serve you, but don't bring us down
161–170 of 255 posts
Re: Let us serve you, but don't bring us down
#162Earlier quoted context omitted.
And if they’re scraping using “serverless” compute? It’s distributed, but could be used without “meaning” to attack. That seems to be what happened to Archive (they said AWS, could be ECS or Lambda, idk). FWIW this thread inspired me to implement Cloudflare Turnstile on one of my pages - highly recommend. As compared to reCAPTCHA your users are never wasting their life away clicking on traffic signs, and as compared…
Extremely unlikely that was lambda unless the user liked to light money on fire.
Well they were an "AI Startup"....
Re: Let us serve you, but don't bring us down
#163I have a side project that scrapes thousands and thousands of pages of a single website. So as not to piss them off (and so they don't try to block me), my script will take about 6 hours. Between each page fetch it sleeps for a small, random amount of time. It's been working like that for years.
What’s the benefit of waiting a random amount of time between requests?
Some zealous systems will infer a very regular request rate as coming from automated services and block them, no matter how gentle the rate.
Re: Let us serve you, but don't bring us down
#164What an incredibly well versed notice from archive.org! Clear, no assumptions, no irony, no accusations, but only reaching out as they request from us.
Very few people have been more dedicated to the original spirit of service of the World Wide Web than Brewster Kahle. It's as though he's been personally monitoring his server logs for the past 30 years.
Re: Let us serve you, but don't bring us down
#165https://support.aws.amazon.com/#/contacts/report-abuse
It should teach them a lesson to have their AWS account banned.
Re: Let us serve you, but don't bring us down
#166Earlier quoted context omitted.
In my opinion, PoW is the only reliable way, to avoid DDoS attacks. Scraping too much, too quickly is a light form of DDoS, although not intentional. PoW was invented in 2006 exactly for that purpose. The genius of bitcoin (not BTC) is that it provides an organized and practical way, for PoW to be used by everyone on the planet. Some people find it strange, because there is an imaginary token created out of pure noth…
I hate to be a pedant but I see it used wrong a lot. "DDOS" stands for Distributed denial of service, specifically indicating the traffic is coming from many sources (wide range of ip), which is what makes it so hard to defend against. Someone scraping too fast would be performing an unintentional DOS, because they probably arent scraping using a botnet (and if they were they probably do in fact intend to attack)
[dead]
Re: Let us serve you, but don't bring us down
#167Archive.org is a bit of a special case, you need to call them repeatedly to archive a website. They do have a rate limit there, it's pretty aggressive* to the point you could trip it by manually using the site. They must have forgotten to limit the OCR files download. * If they had a better API (a simple non-synchronous API would be enough, one where we could send a list of URLs would be even better), one could have…
I believe you can upload WRAC files to IA and ask them to index the content. Saves them the need to do the archiving and you won't be rate limited on their end.
Re: Let us serve you, but don't bring us down
#168There's no reason for Archive.org to allow any traffic from AWS, Azure or Google Cloud. Archive.org should just block the CIDRs for all those networks. Most traffic that comes from them is scraping traffic and usually malicious.
Re: Let us serve you, but don't bring us down
#169Unpopular opinion: Severely rate limit retrieving the files from the website / HTTP endpoint, and loudly point towards downloading the files via torrents. The torrent protocol was meant to relieve this level of server load in mind.
Torrents have the habit of disappearing when no users keep them alive. It happened to me enough times to be wary of such solution. If there's a way to keep them alive regardless of interest I'm all for it.
This is what "proof of storage" or "proof of data availability" blockchain networks are for. They use economic incentives to continuously pay nodes a small amount to store some data and keep it available, and the cryptographic sampling mechanism ensures that less popular data must remain in the available dataset for nodes to be paid, even if it is rarely requested in full.
Re: Let us serve you, but don't bring us down
#170Earlier quoted context omitted.
Maybe mCaptcha [0] is worth a look. It applies a Proof-of-Work like algorithm (not blockchain-related) which makes it very expensive for scrapers to get data in bulk, but poses least amount of friction to individual users. The project is implemented in Rust and received NGI.eu/NLnet funding. I don't know its state of production-readiness, but Codeberg.org is considering using it (this choice is informed by higher res…
I got interested in mCaptcha, followed your link, but couldn’t find anywhere an example of what the end user would deal with. What kind of PoW are we talking about?
You can find some demos linked on github [1], the simplest being: https://demo.mcaptcha.org/widget/?sitekey=pHy0AktWyOKuxZDzFf...