Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We survived 10k requests/second: Switching to signed asset URLs in an emergency

hardcover.app

Post reply on HN
Print view View original

21–30 of 176 posts

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#21

Post by shortrounddev2 »

paxys wrote: post #7

Quick feedback – you've used the term "signed URL" over 50 times in the post without once explaining what it is or how it works.

Rather than allowing any object on a bucket to be downloaded by its raw URL (i.e: http://mycdn.io/abcdefg.jpeg), the backend service needs to generate a "signed" url, which is a short lived URL that grants the user a single request against that resources (GET, POST, PUT, etc.) (i.e: http://mycdn.io/abcdefg.jpeg?signed={securerandomstring}) So you can only use the URL to download it once, and you need to go through the backend API to generate the presigned URL. This could result in your backend getting hammered but you can also use DDOS protection to prevent 10k requests a second from going through your backend

Theyre also a good way to allow users to upload images to your CDN without having to actually upload that data to your web API backend; you just give the user a presigned PUT request URL and they get a one-time ticket to upload to your bucket

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#23

Post by rsstack »

qaq wrote: post #11

Beauty of cloud :) This could be easily served by a $100/month DO droplet with 0 worries about $.

DO _is_ cloud. Using their droplets compared to someone more sophisticated on GCP is an engineering choice, but both are cloud and both have upsides and downsides, and one needs to understand their needs to make the correct decision both among the different providers and within a provider on the right setup.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#24

Post by frankjr »

flockonus wrote: post #6

Have you considered putting cloudflare or similar CDN with unlimited egress in front of your bucket? Reading your blogpost I don't fully get how the current signing implementation can halt massive downloads, or the "attacker"(?) would just adapt their methods to get the signed URLs first and then proceed to download what they are after anyway?

You cannot just put Cloudflare in front of your Google hosted bucket, that's against CF's terms of service. In order to do that you would have to also host the content itself on Cloudflare R2/Images etc. There used to be also html only restriction but that's no longer the case.

> Next, we got rid of the antiquated HTML vs. non-HTML construct, which was far too broad. Finally, we made it clear that customers can serve video and other large files using the CDN so long as that content is hosted by a Cloudflare service like Stream, Images, or R2.

https://blog.cloudflare.com/updated-tos/

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#25

Post by telotortium »

paxys wrote: post #7

Quick feedback – you've used the term "signed URL" over 50 times in the post without once explaining what it is or how it works.

Until the author fixes the post, this is what they're talking about: https://cloud.google.com/storage/docs/access-control/signed-.... Essentially, it ensures that a URL is invalid unless the server signs it with a secret key controlled by the server, which means that clients can't access your assets just by guessing the URL. In addition to signing the URL, the signature can contain metadata such as permissions and expiration time.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#26

Post by tayo42 »

dyogenez wrote: post #2

Earlier this week someone started hitting our Google Cloud Storage bucket with 10k requests a second... for 7 hours. I realized this while working from a coffee shop and spent the rest of the day putting in place a fix. This post goes over what happened, how we put an a solution in place in hours and how we landed on the route we took. I'm curious to hear how others have solved this same problem – generating authenti…

> I'm curious to hear how others have solved this same problem

I think this is interesting to ask, because I often have problems where I'm almost certain it's been solved before, just people don't bother to write about it. Where can people congregate to discuss questions like this?

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#28

Post by JohnMakin »

frankjr wrote: post #24
flockonus wrote: post #6

Have you considered putting cloudflare or similar CDN with unlimited egress in front of your bucket? Reading your blogpost I don't fully get how the current signing implementation can halt massive downloads, or the "attacker"(?) would just adapt their methods to get the signed URLs first and then proceed to download what they are after anyway?

You cannot just put Cloudflare in front of your Google hosted bucket, that's against CF's terms of service. In order to do that you would have to also host the content itself on Cloudflare R2/Images etc. There used to be also html only restriction but that's no longer the case. > Next, we got rid of the antiquated HTML vs. non-HTML construct, which was far too broad. Finally, we made it clear that customers can serve…

This is absolutely nuts to me and would immediately rule out ever hosting anything on google storage for me

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#29

Post by frankjr »

andrewstuart wrote: post #8

I'm always surprised to read how much money companies are willing to spend on things that can be done for essentially nothing. I had a look at the site - why does this need to run on a major cloud provider at all? Why use VERY expensive cloud storage at 9 cents per gigabyte? Why use very expensive image conversion at $50/month when you can run sharp on a Linux server? I shouldn't be surprised - the world is all in on…

> I use CloudFlare R2 to serve your files - its free.

I mean technically it's not free. It's just that they have a very generous "Forever Free" number of read operations (10M/month, $0.36 per million after).

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#30

Post by ghayes »

frankjr wrote: post #24
flockonus wrote: post #6

Have you considered putting cloudflare or similar CDN with unlimited egress in front of your bucket? Reading your blogpost I don't fully get how the current signing implementation can halt massive downloads, or the "attacker"(?) would just adapt their methods to get the signed URLs first and then proceed to download what they are after anyway?

You cannot just put Cloudflare in front of your Google hosted bucket, that's against CF's terms of service. In order to do that you would have to also host the content itself on Cloudflare R2/Images etc. There used to be also html only restriction but that's no longer the case. > Next, we got rid of the antiquated HTML vs. non-HTML construct, which was far too broad. Finally, we made it clear that customers can serve…

Where is this against the GCP or CloudFlare's TOS?
Post reply on HN