I am sorry but who sees a $100 sudden charge, assumes misconfiguration and just goes about their day without digging deeper right away?
We survived 10k requests/second: Switching to signed asset URLs in an emergency
91–100 of 176 posts
Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency
#92Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency
#93I'm always surprised to read how much money companies are willing to spend on things that can be done for essentially nothing. I had a look at the site - why does this need to run on a major cloud provider at all? Why use VERY expensive cloud storage at 9 cents per gigabyte? Why use very expensive image conversion at $50/month when you can run sharp on a Linux server? I shouldn't be surprised - the world is all in on…
Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency
#94Earlier quoted context omitted.
You cannot just put Cloudflare in front of your Google hosted bucket, that's against CF's terms of service. In order to do that you would have to also host the content itself on Cloudflare R2/Images etc. There used to be also html only restriction but that's no longer the case. > Next, we got rid of the antiquated HTML vs. non-HTML construct, which was far too broad. Finally, we made it clear that customers can serve…
Lots of people do this, so you definitely can do this even if its against CF's terms of service, which is something I can't find evidence of.
Because that sentence doesn’t cut the mustard in a professional setting. That’s a FAFO sentiment.
Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency
#95Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency
#96Earlier quoted context omitted.
Doesn't DO charge $0.01/GB for egress overage? That's $150, not $15k. (Although Hetzner or something would've been even less.)
The formatting ate my math it's 1,440,000TB of transfer per month. (16 x 3 x 30 ). That's $14.4k
Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency
#97So your fix was to move the responsibility to the web server and Redis instance? I guess that works but introduces a whole lot more complexity (you mentioned adding rate limiting) and potential for complete outage in the event a lot of requests for images come in again.
That's my worry too. Our server load for our Rails server hasn't gone up even though our throughput has maxed out at 76k requests/second (which I think is a bunch of people from Hacker News going to the Hardcover homepage and downloading 100 images). I don't like that if Rails goes down our images go down. I'd much prefer to separate these out and show the signed URLs in Next.js and be able to generate them through t…
Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency
#98Do any cloud providers have a sensible default or easy-to-enable mode for “you literally cannot spend one penny until you set specific quotas/limits for each resource you’re allocating”?
No, because surprise runaway costs are their entire business model.
Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency
#99Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency
#100I immediately groaned when I read "public bucket." On AWS you'd put CloudFront in front of the (now-private) bucket as a CDN, then use WAF for rate limiting, bot control, etc. In my experience GCP's services work similarly to AWS, so...is this not possible with GCP, or why wasn't this the setup from the get-go? That's the proper way to do things IMO. Signed URLs I only think of when I think of like, paid content or o…
That said, if you use CF in front of S3 (which you should), anyone with a gigabit connection can easily cost you hundreds of dollars. I know this because I did this to myself accidentally.