Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We survived 10k requests/second: Switching to signed asset URLs in an emergency

hardcover.app

Post reply on HN
Print view View original

61–70 of 176 posts

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#61

Post by 0xbadcafebee »

arcfour wrote: post #36

I immediately groaned when I read "public bucket." On AWS you'd put CloudFront in front of the (now-private) bucket as a CDN, then use WAF for rate limiting, bot control, etc. In my experience GCP's services work similarly to AWS, so...is this not possible with GCP, or why wasn't this the setup from the get-go? That's the proper way to do things IMO. Signed URLs I only think of when I think of like, paid content or o…

Google Cloud makes it insanely difficult/non-obvious what services you should use to solve these problems (or how to use them, because they're always difficult to use). They have a maze of unintuitive product names and sub-products and sub-sub-products, finding them in a UX is ridiculous, there's no useful tips/links/walkthroughs in the wizards, and their docs are terrible. It's like being trapped in the goddamn catacombs of Paris. On AWS, using buckets with CDN, ALB & WAF are obvious and easy, but on GCP it's a quagmire.

The other thing is, AWS WAF was released in 2015, and the Google Cloud Armor WAF feature (the what now?) was released in 2020.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#63

Post by dyogenez »

hypeatei wrote: post #42

So your fix was to move the responsibility to the web server and Redis instance? I guess that works but introduces a whole lot more complexity (you mentioned adding rate limiting) and potential for complete outage in the event a lot of requests for images come in again.

That's my worry too. Our server load for our Rails server hasn't gone up even though our throughput has maxed out at 76k requests/second (which I think is a bunch of people from Hacker News going to the Hardcover homepage and downloading 100 images).

I don't like that if Rails goes down our images go down. I'd much prefer to separate these out and show the signed URLs in Next.js and be able to generate them through the API. I think we'll get there, but that's a bigger change than I could reliably make in a day.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#64

Post by ksnsnsj »

paxys wrote: post #10
Waterluvian wrote: post #4

Do any cloud providers have a sensible default or easy-to-enable mode for “you literally cannot spend one penny until you set specific quotas/limits for each resource you’re allocating”?

No, because surprise runaway costs are their entire business model.

Not really, because those clients will be unhappy and cause trouble.

They like the clients which expand slowly.

So going from $100 to $100k in a month by accident they want to avoid while still being able to go from $1k to $100k in a year

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#65

Post by dyogenez »

Alifatisk wrote: post #22

I can't describe the surprise when I saw RoR being mentioned, that was unexpected but made the article way more exciting to read. Wouldn't this be solved by using Cloudflare R2 though?

That's good to hear Any chance to bring in Ruby.

I'm not familiar with Cloudflare R2, so I'll have to check it out. I do like that we can rate limit based on either User ID requesting an image from the API, or by IP address. I'm not sure how we'd handle segmenting by user id with a CDN (but I'd have to read more to understand if that's a possibility).

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#66

Post by frankjr »

KomoD wrote: post #51
frankjr wrote: post #24

Earlier quoted context omitted.

You cannot just put Cloudflare in front of your Google hosted bucket, that's against CF's terms of service. In order to do that you would have to also host the content itself on Cloudflare R2/Images etc. There used to be also html only restriction but that's no longer the case. > Next, we got rid of the antiquated HTML vs. non-HTML construct, which was far too broad. Finally, we made it clear that customers can serve…

You totally can, just not a "disproportionate percentage"

That's when you're caching a whole page which contains images. The OP is talking about putting the CDN in front of a bucket which doesn't serve anything but images (= 100%).

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#68

Post by upon_drumhead »

Given that you want to be good stewards of book data, have you considered publishing bulk snapshots to archive.org on a set cadence? It would strongly reduce any needs to do any sort of bulk scraping and also ensure that should something happen to your service, the data isn't lost forever.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#69

Post by antihero »

arcfour wrote: post #36

I immediately groaned when I read "public bucket." On AWS you'd put CloudFront in front of the (now-private) bucket as a CDN, then use WAF for rate limiting, bot control, etc. In my experience GCP's services work similarly to AWS, so...is this not possible with GCP, or why wasn't this the setup from the get-go? That's the proper way to do things IMO. Signed URLs I only think of when I think of like, paid content or o…

That said, if you use CF in front of S3 (which you should), anyone with a gigabit connection can easily cost you hundreds of dollars. I know this because I did this to myself accidentally.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#70

Post by dyogenez »

andrewstuart wrote: post #8

I'm always surprised to read how much money companies are willing to spend on things that can be done for essentially nothing. I had a look at the site - why does this need to run on a major cloud provider at all? Why use VERY expensive cloud storage at 9 cents per gigabyte? Why use very expensive image conversion at $50/month when you can run sharp on a Linux server? I shouldn't be surprised - the world is all in on…

If you're able to do that, then you have a huge skill! I'm not much of a devops engineer myself, so I'm leveraging work done by others. My skills are in application design. For hosting I try to rely on what others have built and host there.

If I had your skills then our costs would be much smaller. As it stands now we pay about $700/month for everything - the bulk of it for a 16gb ram / 512gb space database.

Post reply on HN