Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We survived 10k requests/second: Switching to signed asset URLs in an emergency

hardcover.app

Post reply on HN
Print view View original

81–90 of 176 posts

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#81

Post by dyogenez »

languagehacker wrote: post #18

Did this guy just write a blog post about how he completely rewrote a functional feature to save $800? In all seriousness, the devil is in the details around this kind of stuff, but I do worry that doing something not even clever, but just nonstandard, introduces a larger maintenance effort than necessary. Interesting problem, and an interesting solution, but I'd probably rather just throw money at it until it gets t…

If this were a business and someone else's money I'd do the same. This is a bootstrapped side project coming out of my own wallet.

If money wasn't an issue, I'd probably just allow people to download images for free.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#82

Post by Sytten »

I really hope this is not the whole of your code otherwise you have a nice open redirect vulnerability on your hand and possibly a private bucket leak if you don't check which bucket you are signing the request for. Never for the love of security take an URL as input from a user without doing a whole lot of checks and sanitization. And don't expect your language parser to be perfect, Orange Tsai demonstrated they can get confused [1].

[1] https://www.blackhat.com/docs/us-17/thursday/us-17-Tsai-A-Ne...

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#83

Post by languagehacker »

dyogenez wrote: post #81
languagehacker wrote: post #18

Did this guy just write a blog post about how he completely rewrote a functional feature to save $800? In all seriousness, the devil is in the details around this kind of stuff, but I do worry that doing something not even clever, but just nonstandard, introduces a larger maintenance effort than necessary. Interesting problem, and an interesting solution, but I'd probably rather just throw money at it until it gets t…

If this were a business and someone else's money I'd do the same. This is a bootstrapped side project coming out of my own wallet. If money wasn't an issue, I'd probably just allow people to download images for free.

Good point! My POV assumed some amount of revenue generation.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#84

Post by taeric »

dyogenez wrote: post #80
taeric wrote: post #78

I'm confused, isn't this literally the use case for a CDN? Edit: I see this is discussed in other threads.

That would solve some of the problems. If the site was previously behind a CDN with a rate limit, I don't think we would have even had this problem. Given that we have the problem now, and that people already have the non-CDN URLs, we needed a solution that allowed us to roll out something ASAP, while allowing people that use our API to continue using the image URLs they've downloaded.

Makes sense. And kudos on getting a solution that works for you! :D

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#85

Post by written-beyond »

0xbadcafebee wrote: post #61
arcfour wrote: post #36

I immediately groaned when I read "public bucket." On AWS you'd put CloudFront in front of the (now-private) bucket as a CDN, then use WAF for rate limiting, bot control, etc. In my experience GCP's services work similarly to AWS, so...is this not possible with GCP, or why wasn't this the setup from the get-go? That's the proper way to do things IMO. Signed URLs I only think of when I think of like, paid content or o…

Google Cloud makes it insanely difficult/non-obvious what services you should use to solve these problems (or how to use them, because they're always difficult to use). They have a maze of unintuitive product names and sub-products and sub-sub-products, finding them in a UX is ridiculous, there's no useful tips/links/walkthroughs in the wizards, and their docs are terrible. It's like being trapped in the goddamn cata…

Honestly this is exactly how I felt about GCP when I was building something that would be used by millions of people. At that scale it's very easy to shoot yourself in the foot and boy does Google make that easy.

There were so many things that were outright wrong in their documentation that caused me many sleepless nights. Like not recommending using a pool or closing cloudSQL connections in server less functions because they'll be closed automatically when the instance spins down.

Don't get me wrong I had used pools extensively before, and I knew you had to close connections but their docs and examples would explicitly show the connections not being closed, just left for them to close when the instance spins down.

Idk why they never thought that an instance might never spin down if it's getting hammered with requests and you end up with hundreds of open connections over multiple instances until GCP starts killing your requests telling you "out of connections" in a server less instance. The vaguest possible error which after a lot of debugging you understand that you can't have more than 100 open connections on a single function instance, but you were technically never supposed to have more than one open at any given time.

sigh

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#86

Post by dyogenez »

Sytten wrote: post #82

I really hope this is not the whole of your code otherwise you have a nice open redirect vulnerability on your hand and possibly a private bucket leak if you don't check which bucket you are signing the request for. Never for the love of security take an URL as input from a user without doing a whole lot of checks and sanitization. And don't expect your language parser to be perfect, Orange Tsai demonstrated they can…

I left off the method that generates the signed URL. It limits the bucket to a specific one per env and blocks some protected folders and file types. I left that out in case someone used it to find an opening to attack.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#87

Post by qaq »

0xbadcafebee wrote: post #75
qaq wrote: post #47

Earlier quoted context omitted.

Didn't pay attention to transfer figure lets switch DO to CCX43 on Hetzner for $50 more

I think you miss the point of the cloud. It's not supposed to be cheaper. If you want cheap, yeah, run on Hetzner. If you want to deploy a WAF with complex rules to route specific traffic to either a multi-region ALB or a bucket with a WAF built in, and do it in 10 minutes, you use the cloud.

I really don't :) I work on this day in and day out. It just that 90% of the projects don't need any of the above.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#88

Post by the8472 »

ksnsnsj wrote: post #67
the8472 wrote: post #5

The dreaded C10k problem, remaining unsolved to this day.

Unlike the original c10k problem, serving those number of connectioms has now morthed from a technical to an economic problem

I don't think the economics of serving 1Gbit have ever added up to 300$ over two days.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#89

Post by sakopov »

I must be missing something obvious, but what do signed URLs have to do with requests going directly to resources in a bucket instead of a CDN of some sort like Cloudflare? Signed URLs are typically used to provide secure access to a resource in a private bucket. But it seems like it's used as a cache of sorts?

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#90

Post by hereonout2 »

wrs wrote: post #60
dyogenez wrote: post #2

Earlier this week someone started hitting our Google Cloud Storage bucket with 10k requests a second... for 7 hours. I realized this while working from a coffee shop and spent the rest of the day putting in place a fix. This post goes over what happened, how we put an a solution in place in hours and how we landed on the route we took. I'm curious to hear how others have solved this same problem – generating authenti…

It sounds like you had public list access to your bucket, which is always bad. However, you can prevent list access, but keep read access to individual objects public. As long as your object names are unguessable (say, a 16-byte random number), you won’t have the problem you had. I haven’t used Rails since they integrated storage, but gems like Paperclip used to do this for you by hashing the image parameters with a…

This was my understanding of signed URLs also. I was wondering why they needed to be cached, then afterwards wondering why the generation was so slow when I read the 250ms part.
Post reply on HN