Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We survived 10k requests/second: Switching to signed asset URLs in an emergency

hardcover.app

Post reply on HN
Print view View original

131–140 of 176 posts

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#131

Post by kijin »

riedel wrote: post #124
telotortium wrote: post #25

Earlier quoted context omitted.

Until the author fixes the post, this is what they're talking about: https://cloud.google.com/storage/docs/access-control/signed-... . Essentially, it ensures that a URL is invalid unless the server signs it with a secret key controlled by the server, which means that clients can't access your assets just by guessing the URL. In addition to signing the URL, the signature can contain metadata such as permissions and e…

Is there any advantage over JWT other than one can put the token into the URL itself (which is technically also possible with JWT I guess, with the downside that it will be probably exposed in logs, etc.)?

Signed URLs work with dumb storage such as Amazon S3 and Google Cloud Storage.

You might want to use JWT to authenticate someone before handing out a signed URL, though.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#132

Post by inopinatus »

riedel wrote: post #124
telotortium wrote: post #25

Earlier quoted context omitted.

Until the author fixes the post, this is what they're talking about: https://cloud.google.com/storage/docs/access-control/signed-... . Essentially, it ensures that a URL is invalid unless the server signs it with a secret key controlled by the server, which means that clients can't access your assets just by guessing the URL. In addition to signing the URL, the signature can contain metadata such as permissions and e…

Is there any advantage over JWT other than one can put the token into the URL itself (which is technically also possible with JWT I guess, with the downside that it will be probably exposed in logs, etc.)?

The signature is verified by the origin bucket/cdn endpoint, so it's in the format used by the applicable public cloud, without depending on any application-specific cookie or bearer values, and with CORS/CSP interactions you can easily reason about.

Signature values could appear in the logs the cloud provider writes, but won't be in your application logs unless you, for example, specifically write out the Location of a 30x redirect (which would be relatively uncommon), and in any case the ephemerality makes them basically uninteresting.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#133

Post by kijin »

hinkley wrote: post #94
voxic11 wrote: post #40

Earlier quoted context omitted.

Lots of people do this, so you definitely can do this even if its against CF's terms of service, which is something I can't find evidence of.

Is this your private opinion, or your professional one? Because that sentence doesn’t cut the mustard in a professional setting. That’s a FAFO sentiment.

It is indeed FAFO, because Cloudflare doesn't spell out what counts as "a disproportionate percentage" of multimedia assets.

If I serve an HTML page through Cloudflare that loads 100 images, each of which is 10x the size of the HTML page, then 99.9% of my bandwidth usage will be images. Is that a disproportionate percentage? It's entirely at Cloudflare's discretion, they can kick out anyone they don't like.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#134

Post by kijin »

frankjr wrote: post #66
KomoD wrote: post #51

Earlier quoted context omitted.

You totally can, just not a "disproportionate percentage"

That's when you're caching a whole page which contains images. The OP is talking about putting the CDN in front of a bucket which doesn't serve anything but images (= 100%).

You can easily work around that problem by putting the CDN in front of everything, including web pages and API calls, not only your bucket of images.

OP might want to do that anyway, since the attacker will now be hammering their Rails app instead of just the bucket.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#135

Post by cuu508 »

ksnsnsj wrote: post #72
andrewstuart wrote: post #8

I'm always surprised to read how much money companies are willing to spend on things that can be done for essentially nothing. I had a look at the site - why does this need to run on a major cloud provider at all? Why use VERY expensive cloud storage at 9 cents per gigabyte? Why use very expensive image conversion at $50/month when you can run sharp on a Linux server? I shouldn't be surprised - the world is all in on…

I have read this argument before. Of cause you can do everything yourself _but it is not free_ You are missing both development cost and much more importantly opportunity cost If I spent a person year on a cheap run architecture while my competitor spent a person year on a value add feature add, he will win

Depends on what skills you have, but running everything on a single machine rather than messing with multiple cloud services can also be cheaper in development cost.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#136

Post by inopinatus »

I recall reports of cases like this nearly every day at AWS, and that was a decade ago.

It wasn't unusual, for first-time victims at least, that we'd a) waive the fees and b) schedule a solution architect to talk them through using signed URLs or some other mitigation. I have no visibility into current practice either at AWS or GCP but I'd encourage OP to seek billing relief nevertheless, it can't hurt to ask. Sustainable customer growth is the public cloud business model, of which billing surprises are the antithesis.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#137

Post by [deleted] »

paxys wrote: post #10
Waterluvian wrote: post #4

Do any cloud providers have a sensible default or easy-to-enable mode for “you literally cannot spend one penny until you set specific quotas/limits for each resource you’re allocating”?

No, because surprise runaway costs are their entire business model.

[deleted]

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#138

Post by rcarmo »

I had to do a similar thing a decade ago when someone started scraping my site by brute force. At the time I was using CoralCDN already, but my server was getting hammered, so I just started serving up assets with hashed URLs and changing the key every 24h--their scraper was dumb enough to not start again from scratch.

I ended up using the exact same code for sharding, and later to move to a static site with Azure Storage (which lets me use SAS tokens for timed expiry if I want to).

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#139

Post by [deleted] »

paxys wrote: post #10
Waterluvian wrote: post #4

Do any cloud providers have a sensible default or easy-to-enable mode for “you literally cannot spend one penny until you set specific quotas/limits for each resource you’re allocating”?

No, because surprise runaway costs are their entire business model.

[deleted]

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#140

Post by theginger »

frankjr wrote: post #74
voxic11 wrote: post #40

Earlier quoted context omitted.

Lots of people do this, so you definitely can do this even if its against CF's terms of service, which is something I can't find evidence of.

> Cloudflare reserves the right to disable or limit your access to or use of the CDN, or to limit your End Users’ access to certain of your resources through the CDN, if you use or are suspected of using the CDN without such Paid Services to serve video or a disproportionate percentage of pictures, audio files, or other large files. If you're putting the CDN in front of a bucket with nothing but images, you're automa…

The issue would be if you just did it for the bucket but didn't load any web pages through cloudlfare, as long as your website uses cloudlfare, this is a very normal and supported use case. You might potentially run into problems if you used it just for the bucket, the pages (or what ever else) the images in the bucket are normally called from aren't using cloudlfare so cloudlfare just sees your bucket and nothing else, I've never come across people having issues with this but I've only ever seen it for people who have at least some of their normal traffic also going through cloudlfare.
Post reply on HN