Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We survived 10k requests/second: Switching to signed asset URLs in an emergency

hardcover.app

Post reply on HN
Print view View original

101–110 of 176 posts

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#101

Post by hinkley »

busymom0 wrote: post #91

> The previous day I was experimenting with Google Cloud Run, trying to migrate our Next.js staging environment from Vercel to there to save some money. I assumed I misconfigured that service and turned it off and went about my day. I am sorry but who sees a $100 sudden charge, assumes misconfiguration and just goes about their day without digging deeper right away?

Are you one of those devs that mistakenly assumes that you salary constitutes 90% of your cost to the company, when in fact it’s closer to 40%?

You want me to spend an hour trying to save the company $100? We just spent $250. And that’s not the half of it. If the company is expecting me to result in $5 in revenue for every dollar they spend on me, we really just lost out on more than $1000.

I’ve worked many places where we didn’t think about opportunity costs. I’ve also been laid off many times.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#103

Post by JohnMakin »

JohnMakin wrote: post #28
frankjr wrote: post #24

Earlier quoted context omitted.

You cannot just put Cloudflare in front of your Google hosted bucket, that's against CF's terms of service. In order to do that you would have to also host the content itself on Cloudflare R2/Images etc. There used to be also html only restriction but that's no longer the case. > Next, we got rid of the antiquated HTML vs. non-HTML construct, which was far too broad. Finally, we made it clear that customers can serve…

This is absolutely nuts to me and would immediately rule out ever hosting anything on google storage for me

i cant delete this but i misread original comment.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#105

Post by busymom0 »

hinkley wrote: post #101
busymom0 wrote: post #91

> The previous day I was experimenting with Google Cloud Run, trying to migrate our Next.js staging environment from Vercel to there to save some money. I assumed I misconfigured that service and turned it off and went about my day. I am sorry but who sees a $100 sudden charge, assumes misconfiguration and just goes about their day without digging deeper right away?

Are you one of those devs that mistakenly assumes that you salary constitutes 90% of your cost to the company, when in fact it’s closer to 40%? You want me to spend an hour trying to save the company $100? We just spent $250. And that’s not the half of it. If the company is expecting me to result in $5 in revenue for every dollar they spend on me, we really just lost out on more than $1000. I’ve worked many places wh…

The author in another comment posted this which very clearly indicated they are bootstrapping a low cash side project from their own wallet:

> If this were a business and someone else's money I'd do the same. This is a bootstrapped side project coming out of my own wallet. If money wasn't an issue, I'd probably just allow people to download images for free.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#106

Post by dyogenez »

upon_drumhead wrote: post #68

Given that you want to be good stewards of book data, have you considered publishing bulk snapshots to archive.org on a set cadence? It would strongly reduce any needs to do any sort of bulk scraping and also ensure that should something happen to your service, the data isn't lost forever.

I hadn't thought of that, but I love the idea! How's that work?

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#108

Post by chrisweekly »

hinkley wrote: post #94
voxic11 wrote: post #40

Earlier quoted context omitted.

Lots of people do this, so you definitely can do this even if its against CF's terms of service, which is something I can't find evidence of.

Is this your private opinion, or your professional one? Because that sentence doesn’t cut the mustard in a professional setting. That’s a FAFO sentiment.

FAFO: F*ck Around and Find Out

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#109

Post by shortrounddev2 »

taeric wrote: post #76
shortrounddev2 wrote: post #21

Earlier quoted context omitted.

Rather than allowing any object on a bucket to be downloaded by its raw URL (i.e: http://mycdn.io/abcdefg.jpeg ), the backend service needs to generate a "signed" url, which is a short lived URL that grants the user a single request against that resources (GET, POST, PUT, etc.) (i.e: http://mycdn.io/abcdefg.jpeg?signed={securerandomstring} ) So you can only use the URL to download it once, and you need to go through…

Worth calling out that the big benefit is you basically lean on the service provider for streaming the data, without having to form a trust relationship between them and the receiver of the data. That is, the entire point is to not put more compute between the requester and the data. The absolute worst place to be would be to have compute that is streaming from the data provider, so that they can stream to the end us…

Exactly. You can selectively gate access to the data but still have it served from a cdn
Post reply on HN