Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We survived 10k requests/second: Switching to signed asset URLs in an emergency

hardcover.app

Post reply on HN
Print view View original

41–50 of 176 posts

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#41

Post by jazir »

andrewstuart wrote: post #19
BigParm wrote: post #15

Earlier quoted context omitted.

How much does it cost to have an ISP let you do that? What are the barriers generally?

Let you do what? What barriers do you see?

> run your own Linux servers

He might have thought it meant running servers on a home network instead of managing remote Linux servers.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#42

Post by hypeatei »

So your fix was to move the responsibility to the web server and Redis instance? I guess that works but introduces a whole lot more complexity (you mentioned adding rate limiting) and potential for complete outage in the event a lot of requests for images come in again.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#43

Post by dyogenez »

paxys wrote: post #7

Quick feedback – you've used the term "signed URL" over 50 times in the post without once explaining what it is or how it works.

Ohh good catch. Just updated the post with a section mentioning what signed URLs are before jumping into the solution.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#45

Post by [deleted] »

tayo42 wrote: post #26
dyogenez wrote: post #2

Earlier this week someone started hitting our Google Cloud Storage bucket with 10k requests a second... for 7 hours. I realized this while working from a coffee shop and spent the rest of the day putting in place a fix. This post goes over what happened, how we put an a solution in place in hours and how we landed on the route we took. I'm curious to hear how others have solved this same problem – generating authenti…

> I'm curious to hear how others have solved this same problem I think this is interesting to ask, because I often have problems where I'm almost certain it's been solved before, just people don't bother to write about it. Where can people congregate to discuss questions like this?

[deleted]

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#46

Post by daemonologist »

atrus wrote: post #17
qaq wrote: post #11

Beauty of cloud :) This could be easily served by a $100/month DO droplet with 0 worries about $.

Not on DO. ~$100 a month droplet gets you about 5TB of transfer out. They pulled 15TB in 7 hours. That's ~1,440,000 (16 3 30) on overage or about $15k extra.

Doesn't DO charge $0.01/GB for egress overage? That's $150, not $15k. (Although Hetzner or something would've been even less.)

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#47

Post by qaq »

atrus wrote: post #17
qaq wrote: post #11

Beauty of cloud :) This could be easily served by a $100/month DO droplet with 0 worries about $.

Not on DO. ~$100 a month droplet gets you about 5TB of transfer out. They pulled 15TB in 7 hours. That's ~1,440,000 (16 3 30) on overage or about $15k extra.

Didn't pay attention to transfer figure lets switch DO to CCX43 on Hetzner for $50 more

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#48

Post by jsheard »

Alifatisk wrote: post #16
paxys wrote: post #13

Earlier quoted context omitted.

Does DO have free bandwidth? If not how exactly does that solve the problem?

I don't think they have unmetered bandwidth?

They don't, although their overage rates are pretty reasonable compared to the big clouds at 1 cent per gig. It's hard to beat Hetzners 0.1 cents per gig, though.

I'd rather pay pennies for bandwidth than rely on "unmetered" bandwidth which tends to suddenly stop being unmetered if you use it too much.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#49

Post by dyogenez »

arcfour wrote: post #36

I immediately groaned when I read "public bucket." On AWS you'd put CloudFront in front of the (now-private) bucket as a CDN, then use WAF for rate limiting, bot control, etc. In my experience GCP's services work similarly to AWS, so...is this not possible with GCP, or why wasn't this the setup from the get-go? That's the proper way to do things IMO. Signed URLs I only think of when I think of like, paid content or o…

That's a good idea. I probably could've put a CDN in front of this and rate limited there while keeping things public. That might've been faster than using Ruby to be honest. The downside was that our API already shared the non-CDN URLs, so that would leave the problem open for anyone who already had that data.
Post reply on HN