Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let us serve you, but don't bring us down

blog.archive.org

Post reply on HN
Print view View original

141–150 of 255 posts

Re: Let us serve you, but don't bring us down

#141

Post by reaperman »

wtallis wrote: post #68
reaperman wrote: post #39

Earlier quoted context omitted.

The 500ms latency will help a lot if tons of requests are coming in serial. But most likely they are coming in parallel. It’s hard to “throttle” a single isolated request from a lone IP.

So.. make the requests serial. Just dump all the requests coming in over IPv6 into a queue, service that queue at a rate that's higher than the non-bot traffic requires but still low enough to be a problem for bots that can't self-limit to a reasonable rate. And of course, manage that queue intelligently so that you start dropping requests before you run out of RAM.

That just sounds like DDoSing yourself? It costs bots almost nothing to wait in the queue, and exactly zero real IPv6 clients will wait 10+ seconds to complete the request.

Re: Let us serve you, but don't bring us down

#142

Post by thangngoc89 »

tastysandwich wrote: post #33

I have a side project that scrapes thousands and thousands of pages of a single website. So as not to piss them off (and so they don't try to block me), my script will take about 6 hours. Between each page fetch it sleeps for a small, random amount of time. It's been working like that for years.

Yes. I did the same. My side project takes 4-5 days for the whole routine because there are random wait in between requests and only 2 active requests at any given moment.

Re: Let us serve you, but don't bring us down

#143

Post by entropyie »

RobotToaster wrote: post #128
entropyie wrote: post #127

Earlier quoted context omitted.

What about this? https://github.com/FriendlyCaptcha/friendly-lite-server

> This distribution is licensed under a non-commercial source available license

Thanks for the clarification. Indeed it is not open source in the truest sense, but I don't consider source-available to be closed-source. Especially when the main limitation seems to be a no reselling clause. This would not prevent most people from deploying to their SaaS for example. Even having the code available to audit is a plus. Again, perfection is the enemy of good.

I've found that black and white thinking in security is very dangerous, as you often end up with very "secure" controls that have terrible UX, which users bypass completely via byob etc... And pwnage ensues. UX is a primary pillar of security.

Re: Let us serve you, but don't bring us down

#144

Post by ElongatedMusket »

maksimur wrote: post #75
x-complexity wrote: post #26

Unpopular opinion: Severely rate limit retrieving the files from the website / HTTP endpoint, and loudly point towards downloading the files via torrents. The torrent protocol was meant to relieve this level of server load in mind.

Torrents have the habit of disappearing when no users keep them alive. It happened to me enough times to be wary of such solution. If there's a way to keep them alive regardless of interest I'm all for it.

See https://webtorrent.io/faq for some ideas of doing what you're wishing

Re: Let us serve you, but don't bring us down

#146

Post by DannyBee »

tornato7 wrote: post #86
emporas wrote: post #61

Earlier quoted context omitted.

Requiring every user to compute it's own PoW is a terrible idea. It defeats the whole purpose. One's person expensive computation is another person's almost free computation. I commented in the past about it: "At first glance, yes, we can create intentionally expensive computations without relying on a blockchain, that would serve the same purpose. In reality we cannot. Special computer hardware (ASICs) could generat…

Ethereum was pretty successfully ASIC-resistant. Other PoW algorithms are GPU-resistant. Even so, I doubt most spammers are sophisticated enough to create ASICs just to make cheaper requests to this guy's service.

Right - the argument here seems totally silly. The cost barrier to entry would be incredibly high.

Unlike blockchain systems, the implementation details of mcaptcha are also totally hidden, and you don't have to maintain compatibility at all.

Someone creates an asic? Great, you can make it not work anymore without affecting any of your users for real.

10+ million down the drain.

Re: Let us serve you, but don't bring us down

#147

Post by shakesbeard »

luizfelberti wrote: post #145

The fact that IA is so liberal with their scraping policies is laudable, because on the other end of the spectrum lies Wikimedia, where you're lucky if you manage to download their dumps at 500kbps, which makes their dumps pretty much impossible to obtain...

They provide monthly torrents. Can't imagine that to be too slow unless there's literally no one seeding.

Re: Let us serve you, but don't bring us down

#148

Post by DannyBee »

emporas wrote: post #61
8organicbits wrote: post #47

Earlier quoted context omitted.

Really interesting! How efficiently does a web browser compute the PoW? I'm concerned that a bot would use an efficient GPU implementation while real users would run an inefficient JS/webcrypto version.

Requiring every user to compute it's own PoW is a terrible idea. It defeats the whole purpose. One's person expensive computation is another person's almost free computation. I commented in the past about it: "At first glance, yes, we can create intentionally expensive computations without relying on a blockchain, that would serve the same purpose. In reality we cannot. Special computer hardware (ASICs) could generat…

This doesn't make any sense.

ASIC's take tons of time and money to design.

Unlike blockchain, where the PoW algorithm is part of basic compatibility, it's not at all in mcaptcha - the users see a checkbox. The rest is implementation details.

If someone creates an ASIC, you can break it very easily by changing the PoW algorithm a bit, and no users are affected.

Even if someone has the money to keep up with you, which is remarkably expensive, they will be too slow right now.

That would be hard to change

Re: Let us serve you, but don't bring us down

#149

Post by wmanley »

emporas wrote: post #61
8organicbits wrote: post #47

Earlier quoted context omitted.

Really interesting! How efficiently does a web browser compute the PoW? I'm concerned that a bot would use an efficient GPU implementation while real users would run an inefficient JS/webcrypto version.

Requiring every user to compute it's own PoW is a terrible idea. It defeats the whole purpose. One's person expensive computation is another person's almost free computation. I commented in the past about it: "At first glance, yes, we can create intentionally expensive computations without relying on a blockchain, that would serve the same purpose. In reality we cannot. Special computer hardware (ASICs) could generat…

The difference between this use case and bitcoin is algorithm mobility. All bitcoin miners must use the same algorithm and get the same result for it to work. This makes it very hard (read slow) to change the algorithm, so there's time and incentive to design, fabricate and deploy ASICs. For client-server PoW the server can choose a new algorithm whenever the old algorithm becomes ineffective.

Re: Let us serve you, but don't bring us down

#150

Post by tyingq »

pfooti wrote: post #5

I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, b…

If you can detect it reliably, poisoning them with bad answers might be enough to cause them to find an easier target.
Post reply on HN