Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let us serve you, but don't bring us down

blog.archive.org

Post reply on HN
Print view View original

121–130 of 255 posts

Re: Let us serve you, but don't bring us down

#121

Post by krackers »

llui85 wrote: post #99
yyyk wrote: post #65

Archive.org is a bit of a special case, you need to call them repeatedly to archive a website. They do have a rate limit there, it's pretty aggressive* to the point you could trip it by manually using the site. They must have forgotten to limit the OCR files download. * If they had a better API (a simple non-synchronous API would be enough, one where we could send a list of URLs would be even better), one could have…

They have a Google Sheets "API" which I've used and works reasonably well: https://archive.org/services/wayback-gsheets/

This has been broken for the past month (just stuck on waiting for workers for several days), did they fix it?

Re: Let us serve you, but don't bring us down

#122

Post by riedel »

emporas wrote: post #60
rapnie wrote: post #34

Earlier quoted context omitted.

Maybe mCaptcha [0] is worth a look. It applies a Proof-of-Work like algorithm (not blockchain-related) which makes it very expensive for scrapers to get data in bulk, but poses least amount of friction to individual users. The project is implemented in Rust and received NGI.eu/NLnet funding. I don't know its state of production-readiness, but Codeberg.org is considering using it (this choice is informed by higher res…

In my opinion, PoW is the only reliable way, to avoid DDoS attacks. Scraping too much, too quickly is a light form of DDoS, although not intentional. PoW was invented in 2006 exactly for that purpose. The genius of bitcoin (not BTC) is that it provides an organized and practical way, for PoW to be used by everyone on the planet. Some people find it strange, because there is an imaginary token created out of pure noth…

The problem to me seems that it needs to scale with the number of requests. So PoW effectively burns energy.

As the problem with captchas, is rather who profits from them and who could track users, it seems that with an OCR system to be protected, it is most reasonable to actually give OCR tasks to humans. IMHO this is far more sustainable and people would understand the value: there is nothing bad in improving ML in general. Maybe someone could even define a sensible PoW task for OCR but I doubt it...

Re: Let us serve you, but don't bring us down

#123

Post by isoos »

Proposal for web-scrapers to self-coordinate: monitor the response latencies and rate limit yourself based on that.

(1) Limit your load / parallelism: do not use more than 4 threads, and if the site takes more than a few seconds to respond, use fewer.

(2) Limit aggregate load: after each request, sleep/wait at least the amount that the previous request took to get served.

(3) If you need more than that, ask the site owners for direct channel.

This way if multiple crawler happen to crawl the same resource-limited site, the site may have a fighting chance.

Re: Let us serve you, but don't bring us down

#124

Post by yyyk »

emporas wrote: post #61
8organicbits wrote: post #47

Earlier quoted context omitted.

Really interesting! How efficiently does a web browser compute the PoW? I'm concerned that a bot would use an efficient GPU implementation while real users would run an inefficient JS/webcrypto version.

Requiring every user to compute it's own PoW is a terrible idea. It defeats the whole purpose. One's person expensive computation is another person's almost free computation. I commented in the past about it: "At first glance, yes, we can create intentionally expensive computations without relying on a blockchain, that would serve the same purpose. In reality we cannot. Special computer hardware (ASICs) could generat…

>One's person expensive computation is another person's almost free computation

We have pretty good guides to what's an expensive computation to everyone. That's how password-hashing algorithms work, much better than BitCoin does. Besides, the existence of specialized methods of compute is hardly determinative. We're not trying to stop TLAs. It just needs to be expensive enough to stop 99%, and at most we'll later update again.

Re: Let us serve you, but don't bring us down

#125

Post by jakear »

RugnirViking wrote: post #117
emporas wrote: post #60

Earlier quoted context omitted.

In my opinion, PoW is the only reliable way, to avoid DDoS attacks. Scraping too much, too quickly is a light form of DDoS, although not intentional. PoW was invented in 2006 exactly for that purpose. The genius of bitcoin (not BTC) is that it provides an organized and practical way, for PoW to be used by everyone on the planet. Some people find it strange, because there is an imaginary token created out of pure noth…

I hate to be a pedant but I see it used wrong a lot. "DDOS" stands for Distributed denial of service, specifically indicating the traffic is coming from many sources (wide range of ip), which is what makes it so hard to defend against. Someone scraping too fast would be performing an unintentional DOS, because they probably arent scraping using a botnet (and if they were they probably do in fact intend to attack)

And if they’re scraping using “serverless” compute? It’s distributed, but could be used without “meaning” to attack. That seems to be what happened to Archive (they said AWS, could be ECS or Lambda, idk).

FWIW this thread inspired me to implement Cloudflare Turnstile on one of my pages - highly recommend. As compared to reCAPTCHA your users are never wasting their life away clicking on traffic signs, and as compared to mCaptcha you don’t have to host the server yourself.

Re: Let us serve you, but don't bring us down

#126

Post by bramd »

RockRobotRock wrote: post #95
rapnie wrote: post #34

Earlier quoted context omitted.

Maybe mCaptcha [0] is worth a look. It applies a Proof-of-Work like algorithm (not blockchain-related) which makes it very expensive for scrapers to get data in bulk, but poses least amount of friction to individual users. The project is implemented in Rust and received NGI.eu/NLnet funding. I don't know its state of production-readiness, but Codeberg.org is considering using it (this choice is informed by higher res…

hCaptcha's accessibility model seems quite good to me: https://www.hcaptcha.com/accessibility

I disagree. To "solve" their CAPTCHAs I had to register by providing a working email address. I don't encounter HCAPTCHA problems that often, so when I need to solve a new one usually my cookie has expired and I have to reopen the link to get a new one before being able to continue. I just store such links in my password manager, but imagine you have to find an email with the link they sent you over a year ago before being able to just continue on with what you were doing. And even then, depending on ad blocking/privacy settings their cookie may not even work.

I think this whole thing is a big hurdle just because I'm unable to solve visual puzzles. Besides, having a company collecting email addresses of people who are disabled in one way or another and giving them an identifying cookie is a privacy/data disaster waiting to happen.

That being said, I think the audio alternatives for visual CAPTCHAs are also unacceptable. Even if you can hear them, they may be hard to solve especially if they are not provided in your mother tongue. I think we can and should be able to do better by now

Re: Let us serve you, but don't bring us down

#127

Post by entropyie »

RobotToaster wrote: post #115
entropyie wrote: post #94

Earlier quoted context omitted.

Security is not all or nothing. There are many applications where adding a small bit of friction in the form of compute will stop 99.9% of abusive traffic. Visual captchas are a plague on the internet, but so is Blockchain mania. I added FriendlyCaptcha to some of my sites, and stopped 100% of abusive traffic. Open source, user friendly, accessible to people with disabilities. https://friendlycaptcha.com/ Most of us…

Friendlycaptcha isn't open source, only the widget/client is. The sever seems to be closed source.

What about this? https://github.com/FriendlyCaptcha/friendly-lite-server

Re: Let us serve you, but don't bring us down

#128

Post by RobotToaster »

entropyie wrote: post #127
RobotToaster wrote: post #115

Earlier quoted context omitted.

Friendlycaptcha isn't open source, only the widget/client is. The sever seems to be closed source.

What about this? https://github.com/FriendlyCaptcha/friendly-lite-server

>This distribution is licensed under a non-commercial source available license

Re: Let us serve you, but don't bring us down

#129

Post by yyyk »

emporas wrote: post #90
8organicbits wrote: post #82

Earlier quoted context omitted.

Looking cross-comment I think you're advocating for micro-payments or micro-proof-of-burn. This is better than having every user compute their own PoW as they'd purchase a small amount of cryptocurrency from an ASIC farm, which should be cheaper/faster/more efficient than using their local hardware. I mostly agree with the concept, but I don't think any such API exists yet with enough adoption to consider this a real…

You are right in most of your points. Today it doesn't exist an API for micro_proof_of_burn. Economic incentives however require a little patience because they need to work their way through the system. 0.0001 BTC is hugely expensive. A blockchain with millionth of a cent transaction, is certainly possible, and it will exist in 2-3 years approximately. In my calculations, with millionth of a cent per transaction, eve…

Micropayment are 'profitable' save for anti-money-laundering laws (not to speak of anti-terrorism laws) which make any such payment have significant compliance costs. The only way to bypass that would be with centralized verification agencies to take liability (and demonetize anyone deemed controversial). Since they're centralized, they'll have zero use for blockchain (good riddance). Does anyone want that future?

Re: Let us serve you, but don't bring us down

#130

Post by marginalia_nu »

RugnirViking wrote: post #117
emporas wrote: post #60

Earlier quoted context omitted.

In my opinion, PoW is the only reliable way, to avoid DDoS attacks. Scraping too much, too quickly is a light form of DDoS, although not intentional. PoW was invented in 2006 exactly for that purpose. The genius of bitcoin (not BTC) is that it provides an organized and practical way, for PoW to be used by everyone on the planet. Some people find it strange, because there is an imaginary token created out of pure noth…

I hate to be a pedant but I see it used wrong a lot. "DDOS" stands for Distributed denial of service, specifically indicating the traffic is coming from many sources (wide range of ip), which is what makes it so hard to defend against. Someone scraping too fast would be performing an unintentional DOS, because they probably arent scraping using a botnet (and if they were they probably do in fact intend to attack)

In practice they're often using botnets. It's what makes this such a difficult problem to deal with. If they weren't it would be trivial to rate-limit or block problematic IPs or subnets.
Post reply on HN