Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let us serve you, but don't bring us down

blog.archive.org

Post reply on HN
Print view View original

21–30 of 255 posts

Re: Let us serve you, but don't bring us down

#21

Post by jb_gericke »

pfooti wrote: post #5

I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, b…

Why not put auth on the endpoint and enforce quotas and rate limiting (an api gateway like kong could handle this for you).

Re: Let us serve you, but don't bring us down

#22

Post by slivanes »

akiselev wrote: post #17
Taek wrote: post #14

Earlier quoted context omitted.

I don't think that works if your attacker has millions of IPs and is only using 200 per second.

How are they getting millions of ipv4 addresses? IIUC that’s at least the equivalent of a /12 block. Do those shady residential proxies really operate at that scale? If they’re ipv6 address wouldn’t they be safe to block across large ranges?

My guess through compromised boxes (or some apps) that act as proxies.

Re: Let us serve you, but don't bring us down

#24

Post by throwawayinrage »

TekMol wrote: post #3

What is the legal ground for archive.org to copy websites? Shouldn't copyright forbid that? They don't even respect robots.txt. So content creators can't even opt out of that. Not that copyright would have copyright holders having to opt out of copying in the first place. How have they not been sued out of existance yet?

[dead]

Re: Let us serve you, but don't bring us down

#25

Post by akiselev »

p-e-w wrote: post #8
radq wrote: post #2

Sounds like they don't have rate limiting implemented? That seems odd, and it's also surprising it isn't talked about in the post.

Indeed. Tens of thousands of requests per second from just 64 hosts? So they allow individual hosts to make hundreds of requests per second, sustained? That sounds crazy. Even for a burst limit hundreds per second would be extremely high.

Archive.org is a core utility for the web to the point where Wikipedia and many other sites would collapse without it in the sense that many if not most of their outbound links would be dead forever. I’m pretty sure it would even impact the US justice system [1].

Obviously judges aren’t going to have to worry about reasonable rate limits but if these DDoSes are rare, I’d much rather they dealt with them on a case by case basis. Without some complex dynamic rate limit that scales based on available compute and demand, rate limiting would be a blunt solution that will necessarily generate false positives.

[1] https://www.theregister.com/2018/09/04/wayback_machine_legit...

Re: Let us serve you, but don't bring us down

#27

Post by reaperman »

akiselev wrote: post #17
Taek wrote: post #14

Earlier quoted context omitted.

I don't think that works if your attacker has millions of IPs and is only using 200 per second.

How are they getting millions of ipv4 addresses? IIUC that’s at least the equivalent of a /12 block. Do those shady residential proxies really operate at that scale? If they’re ipv6 address wouldn’t they be safe to block across large ranges?

It is trivially easy to get millions of IPv6’s, even spread out across a thousand ranges.

It is also trivially “easy” to get past reCAPTCHA, but it costs more. My guess is that a domain name checker tool isn’t worth the cost per request to bypass reCAPTCHA (approximate 0.02 cents per session)

Re: Let us serve you, but don't bring us down

#28

Post by reaperman »

hattmall wrote: post #13
pfooti wrote: post #5

I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, b…

Captcha seems like overkill. Were you not able to implement a JS fingerprinting / bot check before captcha.

Most of the time, the new CAPTCHAs do that and then never show an actual interactive element. The interactive gimmicks are a fallback in cases of high uncertainty.

The webmaster doesn’t need to worry about it, the anti-bot services handle who gets what difficulty of challenge. But the webmaster can specify whether they’d like to be more or less strict/difficult than usual.

Re: Let us serve you, but don't bring us down

#29

Post by boulos »

pfooti wrote: post #5

I run a system at my employer that occasionally gets scraped by malicious users. It can be used to infer the purchasability of a specific domain, which is a moderately-interesting API endpoint, since that requires talking to domain registries. For a while, nobody cared enough about it to abuse the endpoint. But then we started getting about 40 QPS of traffic. We normally get less than 1. I was keeping an eye on it, b…

Rate limit and return a 429?

Re: Let us serve you, but don't bring us down

#30

Post by userbinator »

x-complexity wrote: post #26

Unpopular opinion: Severely rate limit retrieving the files from the website / HTTP endpoint, and loudly point towards downloading the files via torrents. The torrent protocol was meant to relieve this level of server load in mind.

and loudly point towards downloading the files via torrents.

By "loudly", perhaps actually redirect to the .torrent instead, for those who trip the rate limit?

Post reply on HN