Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let us serve you, but don't bring us down

blog.archive.org

Post reply on HN
Print view View original

201–210 of 255 posts

Re: Let us serve you, but don't bring us down

#201

Post by marginalia_nu »

lagniappe wrote: post #172
marginalia_nu wrote: post #120

Earlier quoted context omitted.

Yeah I'm getting this shit too with Marginalia Search. I'm getting about 2-2.5M queries per day that are definitely from bots that would 100% sink my server if they somehow go through the bot mitigation. It peaks at hundreds of search queries per second. To be clear these are search queries, and search queries typically trigger disk reads of about ~10-20 Mb. I get about 20,000 queries per day that may be human.

does your server have its own isp connection or is that your home connection

I'm on my home connection. Not really anywhere near saturating it.

Re: Let us serve you, but don't bring us down

#202

Post by kmeisthax »

smallnamespace wrote: post #182
kmeisthax wrote: post #179

Earlier quoted context omitted.

No. In fact, using such a well-known proof of work function would defeat the purpose of using PoW. Instead of having something that's cheap for occasional users but expensive for spammers, you'd have something that's cheap for people who own mining ASICs and expensive for occasional users. Hardware specialization breaks the economics behind a CAPTCHA. To fight that you need to use a PoW that hasn't been ASIC'd yet, a…

> cheap for people who own mining ASICs and expensive for occasional users. Seems like it should be the same cost (barring the friction of having a wallet, etc.) for both sets of people since Bitcoin is just a commodity and the value is the same to everyone, miner or not. For example, a miner should value some fraction of a BTC the same way anyone else does, since they can sell or buy it at the same price a normal us…

Miners have ASICs - specialized silicon that ONLY mines Bitcoin, but does so many times faster than a CPU or GPU can. The average low-demand user does not have such hardware. Furthermore, because Bitcoin increases difficulty to maintain a fixed transaction rate, ASICs have to be replaced with newer models every few years. So older ASICs are going to be cheaper, but they might still be worth buying if you're a spammer.

So in order to moderately inconvenience said spammer, you have to make each and every ordinary user wait hours mining a few satoshis' worth of hashes in order to be let in. This is the exact opposite of what you want.

Re: Let us serve you, but don't bring us down

#203

Post by that_guy_iain »

tastysandwich wrote: post #33

I have a side project that scrapes thousands and thousands of pages of a single website. So as not to piss them off (and so they don't try to block me), my script will take about 6 hours. Between each page fetch it sleeps for a small, random amount of time. It's been working like that for years.

I build a system that scraped GitHub. Even though GitHub can clearly handle lots of traffic I still rate limited the hell out of it. The only time I've ever seen scrapers get banned is when go super fast. Unless it's LinkedIn and Instagram who guard their product aka data as much as possible.

Re: Let us serve you, but don't bring us down

#204

Post by coldpie »

Lalabadie wrote: post #163
perryprog wrote: post #155

Earlier quoted context omitted.

What’s the benefit of waiting a random amount of time between requests?

Some zealous systems will infer a very regular request rate as coming from automated services and block them, no matter how gentle the rate.

Is this speculation, or something that has actually been seen?

Re: Let us serve you, but don't bring us down

#205

Post by costco »

Why are there always super obnoxious scrapers out there? It doesn't even make sense from the scrapers perspective because it raises the chance of being blocked and having annoying anti bot measures introduced. It especially makes no sense in this case because archive.org would be happy to give them the data for probably less than whatever they were going to spend on AWS. There's no manners anymore.

Re: Let us serve you, but don't bring us down

#206

Post by klysm »

crazygringo wrote: post #196

Is there an open-source rate limiter that works well for sites large and small? It just strikes me as surprising that sites are still dealing with problems like this in 2023. The idea that a site is still manually having to identify a set of IP addresses and block them with human intervention seems absolutely archaic by this point. And I don't think different sites have particularly different needs here... basic patt…

I think rate limiting is the wrong approach. If there’s too much load you need to manage the queue

Re: Let us serve you, but don't bring us down

#207

Post by lastangryman »

syngrog66 wrote: post #200

reminds me of a general pattern I've observed in life: 1. whatever interface you expose to the public, some will abuse it 2. rules/laws/countermeasures are then added and enforced, often universally (though sometimes inconsistently) 3. which then bites, annoys, insults or otherise burdens or adds to the prices paid by ALL the OTHER non-abusing users pf that same interface always. eventually. every time

It's a digital Tragedy of the Commons: https://en.m.wikipedia.org/wiki/Tragedy_of_the_commons

Re: Let us serve you, but don't bring us down

#208

Post by thayne »

crazygringo wrote: post #196

Is there an open-source rate limiter that works well for sites large and small? It just strikes me as surprising that sites are still dealing with problems like this in 2023. The idea that a site is still manually having to identify a set of IP addresses and block them with human intervention seems absolutely archaic by this point. And I don't think different sites have particularly different needs here... basic patt…

It isn't that hard to set up naive rate limiting per ip address. It's a few lines in haproxy, and there is documentation on how to do it. There are a couple of problems that make it more complicated though. The first is that with NATs, you can have a lot of users behind a single IP address, which can result in legitimate requests getting blocked. The second is that, while it can help against a DoS, it doesn't help that much against a DDoS, because the requests are coming from a large number of distinct IP addreesses.

Re: Let us serve you, but don't bring us down

#209

Post by crazygringo »

thayne wrote: post #208
crazygringo wrote: post #196

Is there an open-source rate limiter that works well for sites large and small? It just strikes me as surprising that sites are still dealing with problems like this in 2023. The idea that a site is still manually having to identify a set of IP addresses and block them with human intervention seems absolutely archaic by this point. And I don't think different sites have particularly different needs here... basic patt…

It isn't that hard to set up naive rate limiting per ip address. It's a few lines in haproxy, and there is documentation on how to do it. There are a couple of problems that make it more complicated though. The first is that with NATs, you can have a lot of users behind a single IP address, which can result in legitimate requests getting blocked. The second is that, while it can help against a DoS, it doesn't help th…

But it's precisely those complicated parts though that seem like they should be solved by now. The solution is both of the problems is for when traffic gets unexpectedly high either behind a NAT or globally, everybody gets a CATPCHA when gives you a token and then you continue to rate-limit each token. (If users log in then each login already does this, no CAPTCHA needed.) This strategy is basically what e.g. CloudFlare does with the CAPTCHAs.

Obviously if someone is attempting a large-scale DDoS your servers can't handle it and you'll be using CloudFlare for its scale. But otherwise, for basic protection against greedy spiders who are even trying to evade detection across a ton of VPN/cloud IP's, this strategy works fine. It's exactly the kind of thing that I would expect any large website to implement.

If there isn't an open-source tool that does this, I wonder why not. Or if there is, I wonder why IA isn't using something like it. But heck, IA wasn't even using a simple version -- it was just 64 IP addresses where basic rate-limiting would have worked fine.

Re: Let us serve you, but don't bring us down

#210

Post by [deleted] »

klysm wrote: post #206
crazygringo wrote: post #196

Is there an open-source rate limiter that works well for sites large and small? It just strikes me as surprising that sites are still dealing with problems like this in 2023. The idea that a site is still manually having to identify a set of IP addresses and block them with human intervention seems absolutely archaic by this point. And I don't think different sites have particularly different needs here... basic patt…

I think rate limiting is the wrong approach. If there’s too much load you need to manage the queue

[deleted]
Post reply on HN