Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let us serve you, but don't bring us down

blog.archive.org

Post reply on HN
Print view View original

61–70 of 255 posts

Re: Let us serve you, but don't bring us down

#61

Post by emporas »

8organicbits wrote: post #47
rapnie wrote: post #34

Earlier quoted context omitted.

Maybe mCaptcha [0] is worth a look. It applies a Proof-of-Work like algorithm (not blockchain-related) which makes it very expensive for scrapers to get data in bulk, but poses least amount of friction to individual users. The project is implemented in Rust and received NGI.eu/NLnet funding. I don't know its state of production-readiness, but Codeberg.org is considering using it (this choice is informed by higher res…

Really interesting! How efficiently does a web browser compute the PoW? I'm concerned that a bot would use an efficient GPU implementation while real users would run an inefficient JS/webcrypto version.

Requiring every user to compute it's own PoW is a terrible idea. It defeats the whole purpose. One's person expensive computation is another person's almost free computation.

I commented in the past about it:

"At first glance, yes, we can create intentionally expensive computations without relying on a blockchain, that would serve the same purpose. In reality we cannot. Special computer hardware (ASICs) could generate much cheaper PoW annotations than general purpose computers, and sell it to spammers. Blockchain economic incentives ensure that ASICs will be used by the miners first and foremost."

https://news.ycombinator.com/item?id=33533389

Re: Let us serve you, but don't bring us down

#62

Post by pfooti »

Szpadel wrote: post #57
pfooti wrote: post #5

I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, b…

I had similar issue for one of our clients. My strategy to not affect legit users was to enable mitigations if global traffic crossed some threshold. eg. in your case this could mean if traffic is above eg. 75QPS then captcha is enabled, and if it's below that it's disabled. I don't know what tech stack you are using, but nice trick that i figured out was to abuse rate limiting to detect global traffic (doing if bran…

That is more or less what I landed on myself. (Not quite, but similar reactive configs based on traffic thresholds).

For a while, we had to just set off pagers when global traffic exceeded a threshold and manually toggle the extra hardening, but eventually it became a lot more reactive.

Re: Let us serve you, but don't bring us down

#63

Post by comex »

emporas wrote: post #61
8organicbits wrote: post #47

Earlier quoted context omitted.

Really interesting! How efficiently does a web browser compute the PoW? I'm concerned that a bot would use an efficient GPU implementation while real users would run an inefficient JS/webcrypto version.

Requiring every user to compute it's own PoW is a terrible idea. It defeats the whole purpose. One's person expensive computation is another person's almost free computation. I commented in the past about it: "At first glance, yes, we can create intentionally expensive computations without relying on a blockchain, that would serve the same purpose. In reality we cannot. Special computer hardware (ASICs) could generat…

> Blockchain economic incentives ensure that ASICs will be used by the miners first and foremost.

So instead, what, you want people to buy BTC and send small amounts of it to websites?

Problem is, given varying income levels across society and across the world, one person's expensive micropayment is another person's almost free micropayment.

Re: Let us serve you, but don't bring us down

#65

Post by yyyk »

Archive.org is a bit of a special case, you need to call them repeatedly to archive a website. They do have a rate limit there, it's pretty aggressive* to the point you could trip it by manually using the site. They must have forgotten to limit the OCR files download.

* If they had a better API (a simple non-synchronous API would be enough, one where we could send a list of URLs would be even better), one could have made a lot less calls.

Re: Let us serve you, but don't bring us down

#66

Post by toyg »

fragmede wrote: post #23

This is a job for HTTP 429!

I recently fought and won 429... On a website ran by the very company I work for. They use a vendor for certain services, and such vendor sells a (terrible) API access for bulk operations, so I whipped out Selenium instead; which meant I, the site "admin", was being throttled with 429. A few pauses here and there, and all was well.

Re: Let us serve you, but don't bring us down

#67

Post by Roark66 »

I just wonder, is archive.org getting any government grant money? If they aren't they should. And I'm not even talking about just the US. All sorts of countries (US, UK, Germany - to name the few) and international organisations like EU pour hundreds of millions into "cultural projects" of very questionable value.

How about they actually fund something really worthy of preservation? Of course it is archive.org role to reach out first. For example EU could fund an archive.org mirror in the EU (with certain throughout etc).

Of course opponents of public/government funding have a very good point in that many organisations when they get public money, they find a way to burn through all of it in a lot less efficient way. This can be mitigated by attaching concrete conditions to the grants. One example is a mirror in a specific location.

Re: Let us serve you, but don't bring us down

#68

Post by wtallis »

reaperman wrote: post #39
jszymborski wrote: post #32

Earlier quoted context omitted.

Right, but this is why the GGP's post suggested throttling IPV6. Or at least that was my understanding.

The 500ms latency will help a lot if tons of requests are coming in serial. But most likely they are coming in parallel. It’s hard to “throttle” a single isolated request from a lone IP.

So.. make the requests serial. Just dump all the requests coming in over IPv6 into a queue, service that queue at a rate that's higher than the non-bot traffic requires but still low enough to be a problem for bots that can't self-limit to a reasonable rate. And of course, manage that queue intelligently so that you start dropping requests before you run out of RAM.

Re: Let us serve you, but don't bring us down

#69

Post by emporas »

comex wrote: post #63
emporas wrote: post #61

Earlier quoted context omitted.

Requiring every user to compute it's own PoW is a terrible idea. It defeats the whole purpose. One's person expensive computation is another person's almost free computation. I commented in the past about it: "At first glance, yes, we can create intentionally expensive computations without relying on a blockchain, that would serve the same purpose. In reality we cannot. Special computer hardware (ASICs) could generat…

> Blockchain economic incentives ensure that ASICs will be used by the miners first and foremost. So instead, what, you want people to buy BTC and send small amounts of it to websites? Problem is, given varying income levels across society and across the world, one person's expensive micropayment is another person's almost free micropayment.

In a comment down below, i made it clear i was not referring to BTC. In the internet there is no centralized registry of naming projects. There are currently three projects in which the communities call themselves bitcoin. Any one of the other two bitcoins support much smaller microtransactions than BTC.

Very true, that micropayments could vary on their relative cheapness across the global population. Let's put a number, is 0.01 cent affordable by most people on the planet, for every http request?

Re: Let us serve you, but don't bring us down

#70

Post by ricardo81 »

pfooti wrote: post #5

I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, b…

Understandable. I've seen services that offer residential IP address proxies for as low as $1/GB. FWIW the particular service in mind actually pays the IP owners who opt into it.

I guess your tool is asking the registries if domains are registered.

Post reply on HN