Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We survived 10k requests/second: Switching to signed asset URLs in an emergency

hardcover.app

Post reply on HN
Print view View original

31–40 of 176 posts

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#31

Post by krzys »

JohnMakin wrote: post #28
frankjr wrote: post #24

Earlier quoted context omitted.

You cannot just put Cloudflare in front of your Google hosted bucket, that's against CF's terms of service. In order to do that you would have to also host the content itself on Cloudflare R2/Images etc. There used to be also html only restriction but that's no longer the case. > Next, we got rid of the antiquated HTML vs. non-HTML construct, which was far too broad. Finally, we made it clear that customers can serve…

This is absolutely nuts to me and would immediately rule out ever hosting anything on google storage for me

It’s Cloudflare’s, which prohibits usage not directly related to hosting web content.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#32

Post by [deleted] »

frankjr wrote: post #24
flockonus wrote: post #6

Have you considered putting cloudflare or similar CDN with unlimited egress in front of your bucket? Reading your blogpost I don't fully get how the current signing implementation can halt massive downloads, or the "attacker"(?) would just adapt their methods to get the signed URLs first and then proceed to download what they are after anyway?

You cannot just put Cloudflare in front of your Google hosted bucket, that's against CF's terms of service. In order to do that you would have to also host the content itself on Cloudflare R2/Images etc. There used to be also html only restriction but that's no longer the case. > Next, we got rid of the antiquated HTML vs. non-HTML construct, which was far too broad. Finally, we made it clear that customers can serve…

[deleted]

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#33

Post by [deleted] »

frankjr wrote: post #29
andrewstuart wrote: post #8

I'm always surprised to read how much money companies are willing to spend on things that can be done for essentially nothing. I had a look at the site - why does this need to run on a major cloud provider at all? Why use VERY expensive cloud storage at 9 cents per gigabyte? Why use very expensive image conversion at $50/month when you can run sharp on a Linux server? I shouldn't be surprised - the world is all in on…

> I use CloudFlare R2 to serve your files - its free. I mean technically it's not free. It's just that they have a very generous "Forever Free" number of read operations (10M/month, $0.36 per million after).

[deleted]

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#34

Post by underwater »

languagehacker wrote: post #18

Did this guy just write a blog post about how he completely rewrote a functional feature to save $800? In all seriousness, the devil is in the details around this kind of stuff, but I do worry that doing something not even clever, but just nonstandard, introduces a larger maintenance effort than necessary. Interesting problem, and an interesting solution, but I'd probably rather just throw money at it until it gets t…

It was $800 so far.

Your point is valid for normal usage patterns where there is a direct relationship between active users and cost. But an attack meant OP’s costs were sky rocketing even though usage was flat.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#35

Post by sroussey »

atrus wrote: post #17
qaq wrote: post #11

Beauty of cloud :) This could be easily served by a $100/month DO droplet with 0 worries about $.

Not on DO. ~$100 a month droplet gets you about 5TB of transfer out. They pulled 15TB in 7 hours. That's ~1,440,000 (16 3 30) on overage or about $15k extra.

I used to have my own half server rack and unlimited bandwidth for $500/mo.

My own machines, of course.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#36

Post by arcfour »

I immediately groaned when I read "public bucket."

On AWS you'd put CloudFront in front of the (now-private) bucket as a CDN, then use WAF for rate limiting, bot control, etc. In my experience GCP's services work similarly to AWS, so...is this not possible with GCP, or why wasn't this the setup from the get-go? That's the proper way to do things IMO.

Signed URLs I only think of when I think of like, paid content or other "semi-public" content.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#38

Post by wordofx »

dyogenez wrote: post #2

Earlier this week someone started hitting our Google Cloud Storage bucket with 10k requests a second... for 7 hours. I realized this while working from a coffee shop and spent the rest of the day putting in place a fix. This post goes over what happened, how we put an a solution in place in hours and how we landed on the route we took. I'm curious to hear how others have solved this same problem – generating authenti…

> I'm curious to hear how others have solved this same problem

Not use Google to start with. And not make S3 buckets public. Must be accessed via CloudFront or CF Signed URLs. Making stuff public is dumb.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#39

Post by blibble »

andrewstuart wrote: post #8

I'm always surprised to read how much money companies are willing to spend on things that can be done for essentially nothing. I had a look at the site - why does this need to run on a major cloud provider at all? Why use VERY expensive cloud storage at 9 cents per gigabyte? Why use very expensive image conversion at $50/month when you can run sharp on a Linux server? I shouldn't be surprised - the world is all in on…

yeah, as a crotchety old unix guy, 10k requests a second was a benchmark 30 years ago on an actual server

today a raspberry pi 5 can do 50k/s with TLS no sweat

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#40

Post by voxic11 »

frankjr wrote: post #24
flockonus wrote: post #6

Have you considered putting cloudflare or similar CDN with unlimited egress in front of your bucket? Reading your blogpost I don't fully get how the current signing implementation can halt massive downloads, or the "attacker"(?) would just adapt their methods to get the signed URLs first and then proceed to download what they are after anyway?

You cannot just put Cloudflare in front of your Google hosted bucket, that's against CF's terms of service. In order to do that you would have to also host the content itself on Cloudflare R2/Images etc. There used to be also html only restriction but that's no longer the case. > Next, we got rid of the antiquated HTML vs. non-HTML construct, which was far too broad. Finally, we made it clear that customers can serve…

Lots of people do this, so you definitely can do this even if its against CF's terms of service, which is something I can't find evidence of.
Post reply on HN