Scaling Clearbit to 2M API requests per day
stackshare.io
Scaling Clearbit to 2M API requests per day
1–10 of 12 posts
Re: Scaling Clearbit to 2M API requests per day
#2> Git push to only one repository.
I'm interested in that aspect. Do you use a single repository for all your code and configuration? Does it also hold the state of your infrastructure (instances, fleet definitions etc)?
Re: Scaling Clearbit to 2M API requests per day
#32M req/day are roughly 23 req/s on average. I understand there might be busier periods, however it doesn't seem to be impressive figures.
Re: Scaling Clearbit to 2M API requests per day
#4Who is clearbit?
Re: Scaling Clearbit to 2M API requests per day
#52M req/day are roughly 23 req/s on average. I understand there might be busier periods, however it doesn't seem to be impressive figures.
Thank you. Using days as your time unit for measuring requests over time is not useful. I'd be more interested in the req/sec numbers during peak.
Re: Scaling Clearbit to 2M API requests per day
#62M req/day = 2000000/24/60/60 = 23 req/s average. With a assumed peak of 10x of avg. this is around 230 req/s.
Hmm. Not something I'd use the word 'scaling' for, even if there is a 1:1 write to DB ratio. With 1000 writes/sec to a DB it gets interesting.
Re: Scaling Clearbit to 2M API requests per day
#72M req/day are roughly 23 req/s on average. I understand there might be busier periods, however it doesn't seem to be impressive figures.
Thank you. Using days as your time unit for measuring requests over time is not useful. I'd be more interested in the req/sec numbers during peak.
even by saying the volume of request at peak time doesn't tell the whole story, we still don't know how heavy are such requests.
Re: Scaling Clearbit to 2M API requests per day
#82M req/day = 2000000/24/60/60 = 23 req/s average. With a assumed peak of 10x of avg. this is around 230 req/s. Hmm. Not something I'd use the word 'scaling' for, even if there is a 1:1 write to DB ratio. With 1000 writes/sec to a DB it gets interesting.
Divide that over 18 machines, too!
I end up doing this with customers. "Well we need a solid hosting system. Our site gets 2 million visits a month!" Me: You know, that'd run alright off my iPAQ. (Actually, with a caching frontend like CloudFlare, it really would.)
Edit: Not to be dissing on their post. I think it's interesting and good that people post such articles. It's just that the sense of scale is off. There was a big article here on some large company, I think it was bitly, and the totals came out pretty low. Especially for the amount of servers. Even Twitter was only peaking at a few KTweets/s a bit ago (granted they do a lot of work per tweet but still).
Re: Scaling Clearbit to 2M API requests per day
#9Maybe I'm missing something but as the other comments have pointed out, this is very low scale. I'm not sure it can even be called at-scale in any sense these days.
Also the architecture seems incredibly fragile and complicated for what it's doing. I get that they've built some integrations and backend processes to assemble this data but the API serving could all run on a single app and database server since it's all just reads.
Re: Scaling Clearbit to 2M API requests per day
#102M req/day = 2000000/24/60/60 = 23 req/s average. With a assumed peak of 10x of avg. this is around 230 req/s. Hmm. Not something I'd use the word 'scaling' for, even if there is a 1:1 write to DB ratio. With 1000 writes/sec to a DB it gets interesting.
Divide that over 18 machines, too! I end up doing this with customers. "Well we need a solid hosting system. Our site gets 2 million visits a month!" Me: You know, that'd run alright off my iPAQ. (Actually, with a caching frontend like CloudFlare, it really would.) Edit: Not to be dissing on their post. I think it's interesting and good that people post such articles. It's just that the sense of scale is off. There w…
" Divide that over 18 machines, too!"
Yes therefor I'd said DB writes, which is the first step stone where you need to think a little in scaling.