Ask HN: How do you keep track of releases/deployments of dozens micro-services?

Have a CI/CD pipeline that does the following:

- unit tests each service

- all services fan-in to a job that builds a giant tar file of source/code artefacts. This includes a metadata file that lists service versions or commit hashes

- this "candidate release" is deployed to a staging environment for automated system/acceptance testing

- it is then optionally deployed to prod once the acceptance tests have passed

My experience with micro-services is code-bases that have prematurely adopted the pattern. Based on this, my advice is as follows...

You can deploy the whole platform and/or refactor to a monolith, and maintain one change log which is simple.

That however has its own downsides, so you should find a balance. If you're having trouble keeping track, perhaps re-organize. I read on one HN article that Amazon had 7k employees before they adopted microservices. The benefits have to outweigh the costs. Sometimes the solution to the problem is taking a step back. without more details its hard to say.

So basically one option is refactor [to a monolith] and re-evaluate the split such that you no longer have this problem. Just throw each repo in a sub-folder & make that your new mono-repo & go from there, it is worth an exploratory refactoring, but not a silver bullet.

We’ve been using our own setup for 4 years now. https://github.com/wballard/starphleet

We have 200 services, counting beta and live test variants. Most of the difficulties vanished once we had declarative versioned control of our service config in the ‘headquarters’ repository.

Not aware of anyone else using this approach.

We built a small internal service that receives updates from the build & deployment scripts we run which then presents us with a html page that shows what branch & commit of everything is deployed (along with the branch and commit of every dependency) where, when and by who. It's totally insecure so it can be trivially spoofed, but it's our V1 for our fleet of golang services and it works well.

Our apps are made by 5-15 (micro)services. I'm not sure if this approach would scale to hundreds of services managed by different teams.

We store the source code for all services in subfolders of the same monorepo (one repo one app). Whenever a change in any service is merged to master, the CI rebuilds _all_ the services and pushes new Docker images to our Docker registry. Thanks to Docker layers, if the source code for a service hasn't changed, the build for that service is super-quick, it just adds a new Docker tag to the _existing_ Docker image.

Then we use the Git commit hash to deploy _all_ services to the desired environment. Again, thanks to Docker layers, containers that haven't changed from the previous tag are recreated instantly because they are cached.

From the CI you can check the latest commit hash that was deployed to any environment, and you can use that commit hash to reproduce that environment locally.

Things that I like:

- the Git commit hash is the single thing you need to know to describe a deployment, and it maps nicely to the state of the codebase at that Git commit.

Things that do not always work:

- if you don't write the Dockerfile in the right way, you end up rebuilding services that haven't changed --> build time increases

- containers for services that haven't changed get stopped and recreated --> short unnecessary downtime, unless you do blue-green