Live data from Hacker News

Scaling Clearbit to 2M API requests per day

stackshare.io

11–12 of 12 posts

Re: Scaling Clearbit to 2M API requests per day

#11

Maybe I'm missing something but as the other comments have pointed out, this is very low scale. I'm not sure it can even be called at-scale in any sense these days. Also the architecture seems incredibly fragile and complicated for what it's doing. I get that they've built some integrations and backend processes to assemble this data but the API serving could all run on a single app and database server since it's all…

Hi manigandham, Harlow here (author of the post). You are absolutely correct; serving 2M API requests from a local data store wouldn't be considered "at-scale" these days.

In hindsight I should have added more information about where the actual work is being done -- I definitely missed the mark on parts of this post.

The lions-share of our work is done in an async fashion. 2M API requests (lookups) turns into 40M+ background jobs. These jobs fetch, aggregate, and scrub data from a number of downstream providers.

Re: Scaling Clearbit to 2M API requests per day

#12

> Git push to only one repository. I'm interested in that aspect. Do you use a single repository for all your code and configuration? Does it also hold the state of your infrastructure (instances, fleet definitions etc)?

We have a Git repo for each of the services, and we have a separate repo for each of the Fleet Unit files.

> Git push to only one repository.

This refers to a per-service repository in our deployer app.

Previously we had to push code to each of the servers running a service. Now we push to the deployer app and it leverages Fleet to distribute the code across the available boxes.

Post reply on HN