Thank you for saying it as 10k requests/second. It makes it way more clear than if you had instead said requests/minute, or worse, requests/day.
We survived 10k requests/second: Switching to signed asset URLs in an emergency
151–160 of 176 posts
Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency
#152Rate limiting (and its important cousin, back-off retries) is an important feature of any service being consumed by an "outside entity". There are many different reasons you'll want rate limiting at every layer of your stack, for every request you have: brute-force resistance, [accidental] DDoS protection, resiliency, performance testing, service quality, billing/quotas, and more. Every important service always event…
Very much what I recommend our teams as well. And you can totally start with something careful. Does a single IP really need 50 requests per second?
Like, sure, I have services at work where the answer is "yes". But I have 10 - 20 times more services for which I could cut that to 5 and still be fine.
Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency
#15310k/s... is that a lot? Computers are insanely fast nowadays..!
You also need to take into account the size of each response, how long your server needs to keep the data in memory (e.g. because of latency, the requester's bandwidth, etc), whether requests to the same file can share a buffer or not, how much data you can be sending at the same time while still being responsive (e.g. without slowing down other responses, causing them to take longer, requiring you to keep those resources in memory for longer, and snowballing from there), ..., stuff like that.
For short text messages, probably not an issue. With larger stuff like images or video, I would be more careful.
Still, even for text-only, if you're using PostgreSQL, by default you have a limit of (I think) 100 parallel connections (or 97, because I think 3 are reserved for superusers), but each connection can only be executing one transaction at a time, so that can quickly become a bottleneck depending on your application and how fast you need to make queries vs how long your queries take to return a response. So then you might need to tune some PostgreSQL settings, or add caching, or some other way to work around the issue.
If you add more services, then you also need to keep in mind the latency between those services.
And so on and so on. So RAM and network would probably become an issue way earlier than CPU in most cases.
TL;DR: "It depends".
Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency
#154Beauty of cloud :) This could be easily served by a $100/month DO droplet with 0 worries about $.
DO _is_ cloud. Using their droplets compared to someone more sophisticated on GCP is an engineering choice, but both are cloud and both have upsides and downsides, and one needs to understand their needs to make the correct decision both among the different providers and within a provider on the right setup.
The billing model for VPSs and real big cloud(TM) providers is very different. This is espeically true for bandwith.
Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency
#155Earlier quoted context omitted.
There is no such thing as unlimited bandwidth. What I'm aware of are services which do not charge extra for egress but severely limit your egress bandwidth (like 10 Gbit peak, 100 Mbit avg) And limiting egress bandwidth is better is better done in the service per client than by the hoster for your system
https://getdeploying.com/reference/data-egress Check this out. You _almost_ use the most expensive service. I think you should expand your awareness. Hetzner for instance doesn't mention anywhere that they throttle your 10gbit uplink, but they limit to 20TB/month, with ~1EUR for every TB over. Seems like you wouldn't even have noticed what you described in your article.
20 TB/month is significantly less than 100 Mbit/s
Sure there are cheaper options then GCS but, there is no one providing unlimited free bandwidth
Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency
#156Earlier quoted context omitted.
Not on DO. ~$100 a month droplet gets you about 5TB of transfer out. They pulled 15TB in 7 hours. That's ~1,440,000 (16 3 30) on overage or about $15k extra.
Didn't pay attention to transfer figure lets switch DO to CCX43 on Hetzner for $50 more
This is fascinating. I've been looking into different providers for a high-bandwidth project.
Are there any providers with free egress?
I used Hetzner many years ago but completely forgot about them.
Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency
#157What I just read is that for the cost of a single 16TB hard drive, they were able to rent a hard drive for 7 hours to stream 16TB, and they still had to devote meaningful engineering resources to avoid the cost overrun. Does anybody here have a success story where AWS was either much cheaper to operate or to develop for (ideally both) than the normal alternatives?
Yeah, I'm confused, too - a $60 server with any decent web server on it should be happy chugging along at 5-15k req/s, right?
In general, yes. My rule of thumb for a basic web server is 100k QPS per physical core on cheap hardware, slowing down if it's doing anything intensive (depending on the nature of the images being requested and how the requests are distributed relative to the disks' layouts, they could have been pegged at the disks' throughput for example), speeding up if you have a particularly light workload or better hardware.
Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency
#158What I just read is that for the cost of a single 16TB hard drive, they were able to rent a hard drive for 7 hours to stream 16TB, and they still had to devote meaningful engineering resources to avoid the cost overrun. Does anybody here have a success story where AWS was either much cheaper to operate or to develop for (ideally both) than the normal alternatives?
They don't use AWS, by the way. This was GCP.
Oops, missed that. The question still stands, but read "AWS" as "AWS or a similar service."
Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency
#159Don't understand why hosting providers charge for egress. Why isn't it free? Doesn't that mean that we don't have an open internet, isn't that against net neutrality?
Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency
#160Have you considered putting cloudflare or similar CDN with unlimited egress in front of your bucket? Reading your blogpost I don't fully get how the current signing implementation can halt massive downloads, or the "attacker"(?) would just adapt their methods to get the signed URLs first and then proceed to download what they are after anyway?
Yup. The only mitigation here is that there is a limit to how many different asset URLs they will be able to generate, but if they want to be malicious they can download the same file over and over again and still make you rack up a huge bill.
This is true. I’d still need a CDN in front of the actual files to prevent that. That’s a takeaway for me from this feedback.