Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We survived 10k requests/second: Switching to signed asset URLs in an emergency

hardcover.app

Post reply on HN
Print view View original

121–130 of 176 posts

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#121

Post by MassPikeMike »

EGreg wrote: post #3

We've designed our system for this very use case. Whether it's on commodity hardware or in the cloud, whether or not it's using a CDN and edge servers, there are ways to "nip things in the bud", as it were, by rejecting requests without a proper signed payload. For example, the value of session ID cookies should actually be signed with an HMAC, and checked at the edge by the CDN. Session cookies that represent a auth…

Thanks for making me aware of X-Accel-Redirect!

The write-up discusses X-Accel-Redirect's functionality in the context of qbix. For me, the two were rather hard to tease apart in that context. So for others who feel that way, I would recommend starting with Grant Horwood's introduction to X-Accel-Redirect at

https://gbh.fruitbat.io/2024/05/12/nginx-serving-private-fil...

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#123

Post by andrewstuart »

BenjiWiebe wrote: post #115
blibble wrote: post #39

Earlier quoted context omitted.

yeah, as a crotchety old unix guy, 10k requests a second was a benchmark 30 years ago on an actual server today a raspberry pi 5 can do 50k/s with TLS no sweat

Can you give me an example of how to do 50k/s with TLS on an rpi? Also what do you use to measure that? I've tried a little with httpd (apache) on an older desktop I use as my home server and got terrible results. I can't remember but it might have been single digit or low double digit rps.

try with caddy

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#124

Post by riedel »

telotortium wrote: post #25
paxys wrote: post #7

Quick feedback – you've used the term "signed URL" over 50 times in the post without once explaining what it is or how it works.

Until the author fixes the post, this is what they're talking about: https://cloud.google.com/storage/docs/access-control/signed-... . Essentially, it ensures that a URL is invalid unless the server signs it with a secret key controlled by the server, which means that clients can't access your assets just by guessing the URL. In addition to signing the URL, the signature can contain metadata such as permissions and e…

Is there any advantage over JWT other than one can put the token into the URL itself (which is technically also possible with JWT I guess, with the downside that it will be probably exposed in logs, etc.)?

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#125

Post by hansvm »

What I just read is that for the cost of a single 16TB hard drive, they were able to rent a hard drive for 7 hours to stream 16TB, and they still had to devote meaningful engineering resources to avoid the cost overrun.

Does anybody here have a success story where AWS was either much cheaper to operate or to develop for (ideally both) than the normal alternatives?

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#126

Post by Dylan16807 »

BenjiWiebe wrote: post #111
the8472 wrote: post #88

Earlier quoted context omitted.

I don't think the economics of serving 1Gbit have ever added up to 300$ over two days.

Some quick googling suggests that a dedicated (DIA) 1gbit connection is around $1k/month, which would be $65ish for 2 days.

At that price it would mostly be overhead for getting a relatively slow connection. You could get 10gbps for a similar price.

1gbps from a hosting provider should be a lot less.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#127

Post by deeebug »

wrs wrote: post #60
dyogenez wrote: post #2

Earlier this week someone started hitting our Google Cloud Storage bucket with 10k requests a second... for 7 hours. I realized this while working from a coffee shop and spent the rest of the day putting in place a fix. This post goes over what happened, how we put an a solution in place in hours and how we landed on the route we took. I'm curious to hear how others have solved this same problem – generating authenti…

It sounds like you had public list access to your bucket, which is always bad. However, you can prevent list access, but keep read access to individual objects public. As long as your object names are unguessable (say, a 16-byte random number), you won’t have the problem you had. I haven’t used Rails since they integrated storage, but gems like Paperclip used to do this for you by hashing the image parameters with a…

> That aside, I’m confused about the 250ms thing. You don’t have to hit a Google API to construct a signed URL. It should just be a signature calculation done locally in your server. [0]

I assume the additional latency is the initial cred fetch from the VM Metadata Service to perform that sign, no?

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#128

Post by lionkor »

hansvm wrote: post #125

What I just read is that for the cost of a single 16TB hard drive, they were able to rent a hard drive for 7 hours to stream 16TB, and they still had to devote meaningful engineering resources to avoid the cost overrun. Does anybody here have a success story where AWS was either much cheaper to operate or to develop for (ideally both) than the normal alternatives?

Yeah, I'm confused, too - a $60 server with any decent web server on it should be happy chugging along at 5-15k req/s, right?

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#130

Post by lionkor »

intelVISA wrote: post #119

10k/s... is that a lot? Computers are insanely fast nowadays..!

No. It's not a lot. 20-30k req/s is easy for serving simple, small files. If you have beefy machine (say, a $50 hetzner rootserver), you get a few TB of storage and unlimited or cheap bandwidth.

8-16 cores can easily(!!!) push this kind of data without even heating up, not sure wtf OP is doing. Well, I know what OP is doing - they fell for the idea that the cloud is more scalable.

The issue with this is that the cloud™ starts "scaling" at the first user, whereas a baremetal server needs to scale when you have saturated a 8-16 core modern CPU, a 1-10 Gb/s NiC, 30-60 GB of RAM. In other words, baremetal needs to scale when you actually run out of hardware resources, which is tens or hundreds of thousands of users later.

Edit: for example, at BeamMP, we run on a few bare metal servers, and serve 22k unique users per day in a multiplayer videogame service. Funded by around 800 people donating.

Post reply on HN