Live data from Hacker News

We survived 10k requests/second: Switching to signed asset URLs in an emergency

hardcover.app

91–100 of 176 posts

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#91
> The previous day I was experimenting with Google Cloud Run, trying to migrate our Next.js staging environment from Vercel to there to save some money. I assumed I misconfigured that service and turned it off and went about my day.

I am sorry but who sees a $100 sudden charge, assumes misconfiguration and just goes about their day without digging deeper right away?

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#93

I'm always surprised to read how much money companies are willing to spend on things that can be done for essentially nothing. I had a look at the site - why does this need to run on a major cloud provider at all? Why use VERY expensive cloud storage at 9 cents per gigabyte? Why use very expensive image conversion at $50/month when you can run sharp on a Linux server? I shouldn't be surprised - the world is all in on…

Looks like a site you could build in WordPress with some custom plugins like ACF and host on a single VPS for the most part.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#94
post #40
post #24

Earlier quoted context omitted.

You cannot just put Cloudflare in front of your Google hosted bucket, that's against CF's terms of service. In order to do that you would have to also host the content itself on Cloudflare R2/Images etc. There used to be also html only restriction but that's no longer the case. > Next, we got rid of the antiquated HTML vs. non-HTML construct, which was far too broad. Finally, we made it clear that customers can serve…

Lots of people do this, so you definitely can do this even if its against CF's terms of service, which is something I can't find evidence of.

Is this your private opinion, or your professional one?

Because that sentence doesn’t cut the mustard in a professional setting. That’s a FAFO sentiment.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#95
post #62
post #9

I don't understand, why wasn't there a CDN in front of the public GCS bucket resources?

While this is normally done due to the reasons mentioned, to me that is a significant downside. Why can't GCS act as a CDN, too?

Because then they can’t sell you two products.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#96
post #71

Earlier quoted context omitted.

Doesn't DO charge $0.01/GB for egress overage? That's $150, not $15k. (Although Hetzner or something would've been even less.)

The formatting ate my math it's 1,440,000TB of transfer per month. (16 x 3 x 30 ). That's $14.4k

Ah thanks I see - if the attack/misuse had kept up for the entire month.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#97

So your fix was to move the responsibility to the web server and Redis instance? I guess that works but introduces a whole lot more complexity (you mentioned adding rate limiting) and potential for complete outage in the event a lot of requests for images come in again.

That's my worry too. Our server load for our Rails server hasn't gone up even though our throughput has maxed out at 76k requests/second (which I think is a bunch of people from Hacker News going to the Hardcover homepage and downloading 100 images). I don't like that if Rails goes down our images go down. I'd much prefer to separate these out and show the signed URLs in Next.js and be able to generate them through t…

I don’t have a ton of use cases for functions where they make great sense, not just fill in a bingo card, but generating access errors cheaply is a big one.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#98
post #10

Do any cloud providers have a sensible default or easy-to-enable mode for “you literally cannot spend one penny until you set specific quotas/limits for each resource you’re allocating”?

No, because surprise runaway costs are their entire business model.

Cloud is the new gym membership.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#100
post #36

I immediately groaned when I read "public bucket." On AWS you'd put CloudFront in front of the (now-private) bucket as a CDN, then use WAF for rate limiting, bot control, etc. In my experience GCP's services work similarly to AWS, so...is this not possible with GCP, or why wasn't this the setup from the get-go? That's the proper way to do things IMO. Signed URLs I only think of when I think of like, paid content or o…

That said, if you use CF in front of S3 (which you should), anyone with a gigabit connection can easily cost you hundreds of dollars. I know this because I did this to myself accidentally.

With WAF simple IP-based rate limiting is very simple & cheap. More complex types of limits aren't too difficult either, but even just deploying that is a few clicks.
Post reply on HN