Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We survived 10k requests/second: Switching to signed asset URLs in an emergency

hardcover.app

Post reply on HN
Print view View original

51–60 of 176 posts

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#51

Post by KomoD »

frankjr wrote: post #24
flockonus wrote: post #6

Have you considered putting cloudflare or similar CDN with unlimited egress in front of your bucket? Reading your blogpost I don't fully get how the current signing implementation can halt massive downloads, or the "attacker"(?) would just adapt their methods to get the signed URLs first and then proceed to download what they are after anyway?

You cannot just put Cloudflare in front of your Google hosted bucket, that's against CF's terms of service. In order to do that you would have to also host the content itself on Cloudflare R2/Images etc. There used to be also html only restriction but that's no longer the case. > Next, we got rid of the antiquated HTML vs. non-HTML construct, which was far too broad. Finally, we made it clear that customers can serve…

You totally can, just not a "disproportionate percentage"

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#52

Post by dyogenez »

flockonus wrote: post #6

Have you considered putting cloudflare or similar CDN with unlimited egress in front of your bucket? Reading your blogpost I don't fully get how the current signing implementation can halt massive downloads, or the "attacker"(?) would just adapt their methods to get the signed URLs first and then proceed to download what they are after anyway?

Putting a CDN in front would prevent this at the bucket level, but then someone could still hit the CDN at 10k requests/second. We could rate limit it there though, which would be nice.

The downside is that people already have the URLs for existing bucket directly. So we'd need to change those either way.

The reason why the attacker couldn't just hit the API to get the signed URLs is due to rate limiting that I go over using the rack-attack ruby gem. Since that's limited to 60/second, that's more like 43k images/day max.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#53

Post by 0xbadcafebee »

Rate limiting (and its important cousin, back-off retries) is an important feature of any service being consumed by an "outside entity". There are many different reasons you'll want rate limiting at every layer of your stack, for every request you have: brute-force resistance, [accidental] DDoS protection, resiliency, performance testing, service quality, billing/quotas, and more.

Every important service always eventually gets rate limiting. The more of it you have, the more problems you can solve. Put in the rate limits you think you need (based on performance testing) and only raise them when you need to. It's one of those features nobody adds until it's too late. If you're designing a system from scratch, add rate limiting early on. (you'll want to control the limit per session/identity, as well as in bulk)

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#54

Post by arcfour »

dyogenez wrote: post #49
arcfour wrote: post #36

I immediately groaned when I read "public bucket." On AWS you'd put CloudFront in front of the (now-private) bucket as a CDN, then use WAF for rate limiting, bot control, etc. In my experience GCP's services work similarly to AWS, so...is this not possible with GCP, or why wasn't this the setup from the get-go? That's the proper way to do things IMO. Signed URLs I only think of when I think of like, paid content or o…

That's a good idea. I probably could've put a CDN in front of this and rate limited there while keeping things public. That might've been faster than using Ruby to be honest. The downside was that our API already shared the non-CDN URLs, so that would leave the problem open for anyone who already had that data.

The bucket is private though, only accessible through the CDN. The old URLs would cease to function. On AWS this is implemented through OAI/OAC, granting the CloudFront distribution access via its own unique principal. AWS has had a baseline security recommendation for years now to disable S3 public access at the account/org level.

Maybe this breaks things, maybe you need to expire some caches, but (forgive me for being blunt, I can't think of a better way to say it) that's the cost of not doing things correctly to begin with.

My first thought as a security engineer when setting something up to be public has always been "how hard could someone hit this, and how much would it cost/affect availability?"

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#55

Post by ksnsnsj »

ezekg wrote: post #14
flockonus wrote: post #6

Have you considered putting cloudflare or similar CDN with unlimited egress in front of your bucket? Reading your blogpost I don't fully get how the current signing implementation can halt massive downloads, or the "attacker"(?) would just adapt their methods to get the signed URLs first and then proceed to download what they are after anyway?

Honestly, I would just move to R2 and save on egress fees even without the CDN. Runaway egress bills are no fun. I saved myself thousands $/mo moving to R2.

What is R2?

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#56

Post by dyogenez »

tayo42 wrote: post #26
dyogenez wrote: post #2

Earlier this week someone started hitting our Google Cloud Storage bucket with 10k requests a second... for 7 hours. I realized this while working from a coffee shop and spent the rest of the day putting in place a fix. This post goes over what happened, how we put an a solution in place in hours and how we landed on the route we took. I'm curious to hear how others have solved this same problem – generating authenti…

> I'm curious to hear how others have solved this same problem I think this is interesting to ask, because I often have problems where I'm almost certain it's been solved before, just people don't bother to write about it. Where can people congregate to discuss questions like this?

Hopefully here. Sometimes the best way to get people to respond is to be wrong. I'm sure I've done a bunch of things wrong.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#57

Post by hypeatei »

BigParm wrote: post #15
andrewstuart wrote: post #8

I'm always surprised to read how much money companies are willing to spend on things that can be done for essentially nothing. I had a look at the site - why does this need to run on a major cloud provider at all? Why use VERY expensive cloud storage at 9 cents per gigabyte? Why use very expensive image conversion at $50/month when you can run sharp on a Linux server? I shouldn't be surprised - the world is all in on…

How much does it cost to have an ISP let you do that? What are the barriers generally?

If you're referring to hosting on a home network, you'll probably be behind CGNAT. Your ISP can give you a dedicated IP but it'll most likely cost something.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#58

Post by ksnsnsj »

ponytech wrote: post #44
qaq wrote: post #11

Beauty of cloud :) This could be easily served by a $100/month DO droplet with 0 worries about $.

I rent a bare metal server for $50/month with unlimited bandwith...

There is no such thing as unlimited bandwidth.

What I'm aware of are services which do not charge extra for egress but severely limit your egress bandwidth (like 10 Gbit peak, 100 Mbit avg)

And limiting egress bandwidth is better is better done in the service per client than by the hoster for your system

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#59

Post by ezekg »

ksnsnsj wrote: post #55
ezekg wrote: post #14

Earlier quoted context omitted.

Honestly, I would just move to R2 and save on egress fees even without the CDN. Runaway egress bills are no fun. I saved myself thousands $/mo moving to R2.

What is R2?

Cloudflare's S3-compatible offering with zero egress fees: https://www.cloudflare.com/developer-platform/r2/

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#60

Post by wrs »

dyogenez wrote: post #2

Earlier this week someone started hitting our Google Cloud Storage bucket with 10k requests a second... for 7 hours. I realized this while working from a coffee shop and spent the rest of the day putting in place a fix. This post goes over what happened, how we put an a solution in place in hours and how we landed on the route we took. I'm curious to hear how others have solved this same problem – generating authenti…

It sounds like you had public list access to your bucket, which is always bad. However, you can prevent list access, but keep read access to individual objects public. As long as your object names are unguessable (say, a 16-byte random number), you won’t have the problem you had.

I haven’t used Rails since they integrated storage, but gems like Paperclip used to do this for you by hashing the image parameters with a secret seed to generate the object name.

Using signed URLs is solving a different problem: making people hit your API at least once a day to get a working GCS URL for the image. It’s not clear if that’s an actual problem, as if people want to enumerate your API (as opposed to your bucket), they can do that with the new system too.

That aside, I’m confused about the 250ms thing. You don’t have to hit a Google API to construct a signed URL. It should just be a signature calculation done locally in your server. [0]

https://cloud.google.com/storage/docs/access-control/signing...

Post reply on HN