Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We survived 10k requests/second: Switching to signed asset URLs in an emergency

hardcover.app

Post reply on HN
Print view View original

161–170 of 176 posts

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#161

Post by 0xbadcafebee »

qaq wrote: post #87
0xbadcafebee wrote: post #75

Earlier quoted context omitted.

I think you miss the point of the cloud. It's not supposed to be cheaper. If you want cheap, yeah, run on Hetzner. If you want to deploy a WAF with complex rules to route specific traffic to either a multi-region ALB or a bucket with a WAF built in, and do it in 10 minutes, you use the cloud.

I really don't :) I work on this day in and day out. It just that 90% of the projects don't need any of the above.

Why do you care whether it "needs" to be done a certain way? You could use Hetzner, or you could use some giant cloud infra. Both get the job done. Why do you care which is used?

You probably care because you don't like waste. But another person might not care so much about waste, and cares more about other things. Like the time it takes to get something done, or the technical ability required, or the ability to find someone else to maintain something, or its flexibility in being changed. Or maybe just the fun of building something with cool toys.

You originally commented with cost being the factor. But if the person building this doesn't mind an additional cost, then they don't need to build it on DO or Hetzner. That said, if they do built it on the cloud, they do need to keep in mind the cost... just like if they built it on DO or Hetzner, they would need to keep in mind how to rebuild the instance when it goes down, how to maintain the instance's OS for security patches, etc.

Both require additional considerations besides the one thing. Your choices just move around the necessary complexity.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#162

Post by dyogenez »

wrs wrote: post #60
dyogenez wrote: post #2

Earlier this week someone started hitting our Google Cloud Storage bucket with 10k requests a second... for 7 hours. I realized this while working from a coffee shop and spent the rest of the day putting in place a fix. This post goes over what happened, how we put an a solution in place in hours and how we landed on the route we took. I'm curious to hear how others have solved this same problem – generating authenti…

It sounds like you had public list access to your bucket, which is always bad. However, you can prevent list access, but keep read access to individual objects public. As long as your object names are unguessable (say, a 16-byte random number), you won’t have the problem you had. I haven’t used Rails since they integrated storage, but gems like Paperclip used to do this for you by hashing the image parameters with a…

Thanks for the comment! Few things to reply to from here.

We didn’t have list access enabled, but someone did get a list of files from our API.

Rails with Paperclip and active storage is amazing. Our front end is in Next.js though, so we have people upload straight from Next to GCS, then we sent the GCS URL to Rails. We don’t do pre-processing of images, so just storing the original is good.

They can still download every image, but they’ll be throttled now and kicked out at the middleware level, or permission denied from GCP. 60/min vs 10k/min.

The signature calculation happening might not be hitting Google in that case. I noticed a long data dump in the console after requesting the signed URL combined with the additional latency and assumed. Maybe it’s just a cryptically difficult calculation like bcrypt and it takes a while. Will have to check, because it’s be great to not need a network reliant call for that.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#163

Post by dyogenez »

dantiberian wrote: post #112
dyogenez wrote: post #2

Earlier this week someone started hitting our Google Cloud Storage bucket with 10k requests a second... for 7 hours. I realized this while working from a coffee shop and spent the rest of the day putting in place a fix. This post goes over what happened, how we put an a solution in place in hours and how we landed on the route we took. I'm curious to hear how others have solved this same problem – generating authenti…

Could you explain more why you were you not able to sign the URLs at request time? Creating an HMAC is very fast.

I’m going to have to look into this today. I assuming generating the URLs hit an API, but if those can happen fast locally that changes things.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#164

Post by dyogenez »

sakopov wrote: post #89

I must be missing something obvious, but what do signed URLs have to do with requests going directly to resources in a bucket instead of a CDN of some sort like Cloudflare? Signed URLs are typically used to provide secure access to a resource in a private bucket. But it seems like it's used as a cache of sorts?

I think you have it right. The signed URLs are a way to giving people an address to the files from our API, then they have call it again to key the keys. I suspect if once we put the files behind a CDN with signed keys, we’ll have even more security here.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#165

Post by dyogenez »

elliot07 wrote: post #107

One suggestion to speed up perf. Use bucket#signed_url instead of file#signed_url, otherwise it's doing an HTTP request to Google every generation.

Thank you! I was wondering where the 250ms of latency was coming from. I’ll change this up today.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#166

Post by dyogenez »

solatic wrote: post #118

Did you try sticking your bucket behind Cloud CDN? Google's documentation is inconsistent, but you do not need to make your bucket public, you can instead grant read access only to Cloud CDN: https://cloud.google.com/cdn/docs/using-signed-cookies#confi... Dangerously incorrect documentation claiming the bucket must be public: https://cloud.google.com/cdn/docs/setting-up-cdn-with-bucket...

This sounds like a solid next step. I’d like to stop storing URLs we don’t control in our DB and share URLs to these images behind a CDN. We could slowly roll that out and update each image url in our database over time with both continuing to work.

I didn’t realize you could do this with a private bucket by granting it access either. That combined with IP throttling at the CDN level might be a good replacement for this and cut out the need for Rails.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#167

Post by dyogenez »

inopinatus wrote: post #136

I recall reports of cases like this nearly every day at AWS, and that was a decade ago. It wasn't unusual, for first-time victims at least, that we'd a) waive the fees and b) schedule a solution architect to talk them through using signed URLs or some other mitigation. I have no visibility into current practice either at AWS or GCP but I'd encourage OP to seek billing relief nevertheless, it can't hurt to ask. Sustai…

I recently had a call with Google and have a sales/solution person I’ve been talking to about moving more services there. I’ll share what happened and see what they say.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#168

Post by dyogenez »

busymom0 wrote: post #105
hinkley wrote: post #101

Earlier quoted context omitted.

Are you one of those devs that mistakenly assumes that you salary constitutes 90% of your cost to the company, when in fact it’s closer to 40%? You want me to spend an hour trying to save the company $100? We just spent $250. And that’s not the half of it. If the company is expecting me to result in $5 in revenue for every dollar they spend on me, we really just lost out on more than $1000. I’ve worked many places wh…

The author in another comment posted this which very clearly indicated they are bootstrapping a low cash side project from their own wallet: > If this were a business and someone else's money I'd do the same. This is a bootstrapped side project coming out of my own wallet. If money wasn't an issue, I'd probably just allow people to download images for free.

We usually spend about $60/month at Google anyways, so $100 wasn’t a crazy jump. That could be one left on Cloud Run instance. When it jumped to $300 total after disabling it that’s when I got worried.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#169

Post by jacobr1 »

riedel wrote: post #124
telotortium wrote: post #25

Earlier quoted context omitted.

Until the author fixes the post, this is what they're talking about: https://cloud.google.com/storage/docs/access-control/signed-... . Essentially, it ensures that a URL is invalid unless the server signs it with a secret key controlled by the server, which means that clients can't access your assets just by guessing the URL. In addition to signing the URL, the signature can contain metadata such as permissions and e…

Is there any advantage over JWT other than one can put the token into the URL itself (which is technically also possible with JWT I guess, with the downside that it will be probably exposed in logs, etc.)?

These came out before JWTs gained broad adoption. They are accomplishing the same technical objective via a different implementation.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#170

Post by ku1ik »

dyogenez wrote: post #163
dantiberian wrote: post #112

Earlier quoted context omitted.

Could you explain more why you were you not able to sign the URLs at request time? Creating an HMAC is very fast.

I’m going to have to look into this today. I assuming generating the URLs hit an API, but if those can happen fast locally that changes things.

Yup, pre-signing is fast and local, without any I/O. It’s just math. You could likely pre-sign thousands of URLs per second if you needed.
Post reply on HN