Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We survived 10k requests/second: Switching to signed asset URLs in an emergency

hardcover.app

Post reply on HN
Print view View original

171–176 of 176 posts

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#171

Post by qaq »

0xbadcafebee wrote: post #161
qaq wrote: post #87

Earlier quoted context omitted.

I really don't :) I work on this day in and day out. It just that 90% of the projects don't need any of the above.

Why do you care whether it "needs" to be done a certain way? You could use Hetzner, or you could use some giant cloud infra. Both get the job done. Why do you care which is used? You probably care because you don't like waste. But another person might not care so much about waste, and cares more about other things. Like the time it takes to get something done, or the technical ability required, or the ability to find…

I generally care about unneeded complexity mainly because often times when you switch projects you inherent a ton of it. Also many projects don't see the light of day because people overcomplicated the infra too much.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#172

Post by BenjiWiebe »

Dylan16807 wrote: post #126
BenjiWiebe wrote: post #111

Earlier quoted context omitted.

Some quick googling suggests that a dedicated (DIA) 1gbit connection is around $1k/month, which would be $65ish for 2 days.

At that price it would mostly be overhead for getting a relatively slow connection. You could get 10gbps for a similar price. 1gbps from a hosting provider should be a lot less.

Ya I was giving a bit of a worst case estimate, for a link with no limiting or quota.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#173

Post by dyogenez »

upon_drumhead wrote: post #114
dyogenez wrote: post #106

Earlier quoted context omitted.

I hadn't thought of that, but I love the idea! How's that work?

Register for an account and create a new item. You can replace files in the item , update the description to indicate what date the snapshot was made and what it contains. https://help.archive.org/help/managing-and-editing-your-item... It's a very open platform. Think up what the best format for your data is and upload a compressed zip file or tar.gz of the data. I'd likely do different archives for images and metada…

Nice! Yeah I like the idea of sharing our static book data here on some kind of cadence. I'll pencil this in by the end of the year.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#174

Post by hinkley »

busymom0 wrote: post #105
hinkley wrote: post #101

Earlier quoted context omitted.

Are you one of those devs that mistakenly assumes that you salary constitutes 90% of your cost to the company, when in fact it’s closer to 40%? You want me to spend an hour trying to save the company $100? We just spent $250. And that’s not the half of it. If the company is expecting me to result in $5 in revenue for every dollar they spend on me, we really just lost out on more than $1000. I’ve worked many places wh…

The author in another comment posted this which very clearly indicated they are bootstrapping a low cash side project from their own wallet: > If this were a business and someone else's money I'd do the same. This is a bootstrapped side project coming out of my own wallet. If money wasn't an issue, I'd probably just allow people to download images for free.

But you’re not the author, and still went with incredulity as your response. My question still stands.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#175

Post by hinkley »

kijin wrote: post #133
hinkley wrote: post #94

Earlier quoted context omitted.

Is this your private opinion, or your professional one? Because that sentence doesn’t cut the mustard in a professional setting. That’s a FAFO sentiment.

It is indeed FAFO, because Cloudflare doesn't spell out what counts as "a disproportionate percentage" of multimedia assets. If I serve an HTML page through Cloudflare that loads 100 images, each of which is 10x the size of the HTML page, then 99.9% of my bandwidth usage will be images. Is that a disproportionate percentage? It's entirely at Cloudflare's discretion, they can kick out anyone they don't like.

I spend a lot of time at work trying to make sure we have our surprises as known unknowns, not unknown unknowns. And that includes avoiding surprises that are entirely in a third party’s control. I’ve heard too many stories of cron jobs shutting people off a midnight. And too many of them shutting you off on the first. Which sometimes comes on a Saturday. If I’m awake at midnight on a Saturday I’m not in a place where I want to deal with a surprise like that.

Give me something with fast feedback, where I can warn myself and others that some serious shit might happen when I push this button. Not random shit at midnight or 00:00 GMT.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#176

Post by flockonus »

dyogenez wrote: post #52
flockonus wrote: post #6

Have you considered putting cloudflare or similar CDN with unlimited egress in front of your bucket? Reading your blogpost I don't fully get how the current signing implementation can halt massive downloads, or the "attacker"(?) would just adapt their methods to get the signed URLs first and then proceed to download what they are after anyway?

Putting a CDN in front would prevent this at the bucket level, but then someone could still hit the CDN at 10k requests/second. We could rate limit it there though, which would be nice. The downside is that people already have the URLs for existing bucket directly. So we'd need to change those either way. The reason why the attacker couldn't just hit the API to get the signed URLs is due to rate limiting that I go ov…

> someone could still hit the CDN at 10k requests/second

CDNs have mechanism to rate limit that you can easily configure, and they will be better at this than a ruby gem (no offence to that).

On Ruby you're taking on the rate limiting job down to your CPU and limited visibility per IP... idk man, cloudflare is 20/month.

Post reply on HN