111–120 of 176 posts
Earlier quoted context omitted.
Unlike the original c10k problem, serving those number of connectioms has now morthed from a technical to an economic problem
I don't think the economics of serving 1Gbit have ever added up to 300$ over two days.
Earlier this week someone started hitting our Google Cloud Storage bucket with 10k requests a second... for 7 hours. I realized this while working from a coffee shop and spent the rest of the day putting in place a fix. This post goes over what happened, how we put an a solution in place in hours and how we landed on the route we took. I'm curious to hear how others have solved this same problem – generating authenti…
We've designed our system for this very use case. Whether it's on commodity hardware or in the cloud, whether or not it's using a CDN and edge servers, there are ways to "nip things in the bud", as it were, by rejecting requests without a proper signed payload. For example, the value of session ID cookies should actually be signed with an HMAC, and checked at the edge by the CDN. Session cookies that represent a auth…
Ohh, using the session ID in the URL would be a nice addition to this. We already generate session tokens for every user - guests and logged in users. We could pass that through to segment on rather than IP address.
Given that you want to be good stewards of book data, have you considered publishing bulk snapshots to archive.org on a set cadence? It would strongly reduce any needs to do any sort of bulk scraping and also ensure that should something happen to your service, the data isn't lost forever.
I hadn't thought of that, but I love the idea! How's that work?
https://help.archive.org/help/managing-and-editing-your-item...
It's a very open platform. Think up what the best format for your data is and upload a compressed zip file or tar.gz of the data.
I'd likely do different archives for images and metadata, so people that want to just process metadata can download that specific data and work on it.
Luckily as you can edit over time, you can experiment and adjust based upon user's feedback.
I'm always surprised to read how much money companies are willing to spend on things that can be done for essentially nothing. I had a look at the site - why does this need to run on a major cloud provider at all? Why use VERY expensive cloud storage at 9 cents per gigabyte? Why use very expensive image conversion at $50/month when you can run sharp on a Linux server? I shouldn't be surprised - the world is all in on…
yeah, as a crotchety old unix guy, 10k requests a second was a benchmark 30 years ago on an actual server today a raspberry pi 5 can do 50k/s with TLS no sweat
I've tried a little with httpd (apache) on an older desktop I use as my home server and got terrible results. I can't remember but it might have been single digit or low double digit rps.
Bots these days are our of control and have lost their mind!
Google's documentation is inconsistent, but you do not need to make your bucket public, you can instead grant read access only to Cloud CDN: https://cloud.google.com/cdn/docs/using-signed-cookies#confi...
Dangerously incorrect documentation claiming the bucket must be public: https://cloud.google.com/cdn/docs/setting-up-cdn-with-bucket...