Ask HN: How do you keep track of releases/deployments of dozens micro-services? • Hacker News Board

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#31

Post by Prefinem » Wed, Jan 17, 2018, 4:03 PM UTC

jrowley wrote: post #30

Prefinem wrote: post #28

Earlier quoted context omitted.

We follow a very similar pattern and we are at over 150 micro services right now (AWS Lambda). A couple of things different that we do since we are building and then deploying to AWS: - Build only on dedicated deployment branches (beta, qa, preview, prod) - Build all functions (transpile, yarn, lint, etc) on every merge into the branch, but only deploy functions with different checksums (saves on api calls to AWS) -…

Just to be clear, all of your functions exist inside one mono-repo, correct? You don't use git submodules at all? Looks like a pretty solid build process. Thanks for the insight!

Yes, we have all of our functions in a single mono repo, broken into projects, and then folders for each function, similar to this:

- src/project1/function1/

- src/project1/function2/

- src/project2/function1/

- src/project3/function1/

- src/project3/function2/

- src/project3/function3/

Deploying the functions is done by project, so we deploy all of one project, then move to the next, and so on and so forth.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#32

Post by dhinus » Wed, Jan 17, 2018, 4:15 PM UTC

iamtew wrote: post #27

dhinus wrote: post #19

Our apps are made by 5-15 (micro)services. I'm not sure if this approach would scale to hundreds of services managed by different teams. We store the source code for all services in subfolders of the same monorepo (one repo one app). Whenever a change in any service is merged to master, the CI rebuilds _all_ the services and pushes new Docker images to our Docker registry. Thanks to Docker layers, if the source code…

> Whenever a change in any service is merged to master, the CI rebuilds _all_ the services and pushes new Docker images to our Docker registry. Why are you rebuilding _all_ the services, wouldn't it make sense to just rebuild the ones that have changes? You're now rebuilding perfectly working services without any new changes just because some other service changed, or am I misunderstanding something here?

Because we want to make sure that in the Docker registry we have _all_ services tagged with the latest commit.

For example you might have a Git history like this:

* 89abcde Fix bug in service_b

* 1234567 Initial commit including service_a and service_b

When 89abcde is pushed, the CI rebuilds both service_a and service_b so we can simply "deploy 89abcde" and you always have only one hash for all services, that is also nicely the same hash of the corresponding Git commit.

The trick to avoid rebuilding perfectly working services is to use Docker layer caching so that when you build service_a (that hasn't changed) Docker skips all steps and simply adds the new tag to the _existing_ Docker image. The second build for service_a should take about 1 second.

In our Docker registry we end up with:

service_a:1234567

service_a:89abcde

service_b:1234567

service_b:89abcde

But the two service_a Docker images are _the same image_, with two different tags.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#33

Post by perlgeek » Wed, Jan 17, 2018, 4:17 PM UTC

We have separate repos for each service, and use https://gocd.org/ to build, test and deploy each separately. But, you could also configure it to only trigger builds from changes in certain directories. There is a single pipeline template from which all pipelines are instantiated.

Independent deployments are one of the key advantages of microservices. If you don't use that feature, why use microservices at all? Just for scalability? Or because it was the default choice?

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#34

Post by praneshp » Wed, Jan 17, 2018, 5:05 PM UTC

dhinus wrote: post #19

Our apps are made by 5-15 (micro)services. I'm not sure if this approach would scale to hundreds of services managed by different teams. We store the source code for all services in subfolders of the same monorepo (one repo one app). Whenever a change in any service is merged to master, the CI rebuilds _all_ the services and pushes new Docker images to our Docker registry. Thanks to Docker layers, if the source code…

Quick question: Does docker, or any other higher level service, let you "tag" images? Ideally, you could build only changed stuff, and use that sha to tag every image. That way you still get the benefits of one hash, and that hash representing the state of the codebase as well, while cutting down on build time.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#35

Post by char_pointer » Wed, Jan 17, 2018, 7:05 PM UTC

https://github.com/ankyra/escape (disclaimer: I'm one of the authors)

We use Escape to version and deploy our microservices across environments and even relate it to the underlying infrastructure code so we can deploy our whole platform as a single unit if needs be.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#36

Post by greendata » Wed, Jan 17, 2018, 7:07 PM UTC

Prefinem wrote: post #31

jrowley wrote: post #30

Earlier quoted context omitted.

Just to be clear, all of your functions exist inside one mono-repo, correct? You don't use git submodules at all? Looks like a pretty solid build process. Thanks for the insight!

Yes, we have all of our functions in a single mono repo, broken into projects, and then folders for each function, similar to this: - src/project1/function1/ - src/project1/function2/ - src/project2/function1/ - src/project3/function1/ - src/project3/function2/ - src/project3/function3/ Deploying the functions is done by project, so we deploy all of one project, then move to the next, and so on and so forth.

That's a great model. Do you use cloudformation each for deployment? If so, have you thought of creating a single cloudformation template for the whole deployment so you can do the entire deployment in one stack update?

Have you encountered any issues to watch out for when only using one APIG for each environment (150 micro-services). Have you encountered any downsides to doing this versus 1 micro-service to 1 APIG? I'm also running into the Gateway throttle limits and I think deploying many micro-services (like you have done) to 1 APIG is the best solution.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#37

Post by char_pointer » Wed, Jan 17, 2018, 7:09 PM UTC

char_pointer wrote: post #35

https://github.com/ankyra/escape (disclaimer: I'm one of the authors) We use Escape to version and deploy our microservices across environments and even relate it to the underlying infrastructure code so we can deploy our whole platform as a single unit if needs be.

Just to add: I've worked on pipelines like this for dozens of clients and I'd be happy to talk more in-depth about your options, as business requirements do tend to influence your delivery pipeline a lot. Email is in my profile if you're interested.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#38

Post by tfjaeckel » Wed, Jan 17, 2018, 8:12 PM UTC

mickeyben wrote: post #2

What do you mean by keep track? Do you want to be aware of deployments? A Slack notification could do it. Or do you want to correlate deployments with other metrics? In this case we instrument our deployments into our monitoring stack (influxdb/grafana) and use this as annotations for the rest of our monitoring. We can also graph the number of releases per project on different aggregates.

I think Slack notifications are really nice to see what's going on right now but not so great to see the state of dozens of service, i.e. what version is deployed to what environment.

Then there is the issue of linking the Git release/tag with the corresponding changes, say from a ticketing system such as Jira. That can be helpful to communicate changes to other people within the organization and to users.

How do you define dependencies for releasing new versions to service? Likely going to happen at some point when you have non-trivial changes to services.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#39

Post by _drFaust » Wed, Jan 17, 2018, 8:44 PM UTC

Got about 80+ services. One repo per service, each service has it's own kubernetes yaml that details the services deploys to the cluster. K8s has a huge ecosystem for monitoring, versioning, health, autoscaling and discovery. On top of that, each repo has a separate slack channel that receives notifications for repo changes, comments, deployments, container builds, datadog monitoring events, etc. There are also core maintainers per repo to maintain consistency.

For anyone that has begun the microservice journey, kubernetes can be intimidating but way worth it. Our original microservice infrastructure was rolled way before k8s and it's just night and day to work with now, the kubernetes team has thought of just about every edge case.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#40

Post by fajran » Wed, Jan 17, 2018, 9:25 PM UTC

dhinus wrote: post #19

Our apps are made by 5-15 (micro)services. I'm not sure if this approach would scale to hundreds of services managed by different teams. We store the source code for all services in subfolders of the same monorepo (one repo one app). Whenever a change in any service is merged to master, the CI rebuilds _all_ the services and pushes new Docker images to our Docker registry. Thanks to Docker layers, if the source code…

At work we also use a monorepo that consists of subfolder of services. We use Kubernetes and we store the config files of each service (and each environment dev/staging/prod) inside the same repo. The k8s config files are placed in directories following this pattern: `/deployment//.yaml`.

To avoid rebuilding all services on every commit, we use Bazel to help determine what services need to be rebuilt. Note that we don't use Bazel as build system but just a tool to see what services are changed -- essentially we only use `filegroup` Bazel rule. After a push to git repo, we basically do (1) `git diff --name-only ` to get changed files, (2) run `bazel query 'rdeps(..., set(list of changed files))'` at both `` and `` commits, and (3) combine the results of `bazel query` and look for the affected services.

Once we know what services need to be rebuilt, we trigger Jenkins jobs of those services. Each service will have its own Jenkins job and Jenkinsfile (we use Pipeline). Here we also package the application as Docker image and push it to the internal registry.

We keep track of what is released using "production" branch for each service. Once we have a build to release, we (1) create a "release candidate" branch from the commit of the build, (2) update the k8s config file, (3) apply the k8s config, and (4) merge this branch to the production branch of the service if everything is ok. Then we merge back the production branch to master branch.