Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you keep track of releases/deployments of dozens micro-services?

news.ycombinator.com

Post reply on HN
Print view View original

111–120 of 140 posts

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#111

Post by lowbloodsugar »

giantg2 wrote: post #51
jayd16 wrote: post #42

If you need to keep track its probably too late. What makes them services is that they should be able to be deployed without a bunch of orchestration of other services. You can solve this by having backwards compatible apis. That said, to know what changes would actually break things you'd ideally have a suite of tests.

"If you need to keep track its probably too late. What makes them services is that they should be able to be deployed without a bunch of orchestration of other services." If only you could tell my bosses/architects that. They won't listen to me. Edit: why downvote?

>If only you could tell my bosses/architects that. They won't listen to me.

Then leave and go somewhere where they will. I wasted too much of my life trying to "change things from within", but I finally learned the lesson. If you have no authority but are held accountable, then GTFO.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#112

Post by nonameiguess »

UK-Al05 wrote: post #50

You've broken the microservice abstraction if this a problem. A team should own a microservice, you release as soon as the team able to. You version your apis, so you don't break any services which rely on yours.

Although this is true, there is still the additional problem that a lot of customers, specifically government customers, require frozen known versions of everything as a requirement for acceptance testing. Auditing standards for the aerospace industry require that what is running in ops is exactly the same version of everything that was running when an acceptance test was witnessed and signed.

It's a totally impractical standard for modern software development, but the developers themselves have no choice in the matter until the customers change.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#113

Post by GordonS »

jjice wrote: post #16
monster_group wrote: post #7

I don't keep track. All microservices use continuous deployment pipelines. If you check in code and it passes all the tests, it will make it out to prod some time in the next few hours.

How do updates to a database work through that pipeline? Do migrations run through and rolled back automatically as needed?

Not the OP, but the way I handle this to to ensure that all migrations are backwards compatible - the current and new versions of the app/API/service must be able to run with the old and new database.

This requires a little discipline, but if you follow a few simple rules it's not really that arduous:

  - when adding a new column, it must have a default value set, or be nullable

  - don't drop any columns

  - don't rename any columns
Now, for those last 2, what I really mean is "don't do it in a single release" - if you want to make destructive changes, do it over the course of 2 releases.

  - release 1: remove dependencies on the column from the app/API/service

  - release 2: performs the database migration with destructive changes
It probably sounds more difficult than it actually is :) In reality, I don't make destructive changes that often though.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#114

Post by 100011_100001 »

JamesSwift wrote: post #28
100011_100001 wrote: post #25

My team is responsible for about 200 microservices being deployed, some of them have 10 or more pods. We don't do continuous delivery. Instead it's done by 5 different groups deploying once every week or two. Our production deployment jobs are in Jenkins and isolated. It's easy to check what was deployed when. We also have a script written that can run an environment report to see what versions and which microservice…

That seems overwhelming to me. Do you feel this is a tenable strategy? Or is the number of deployments becoming an issue?

If you automate enough you almost don't need to know the specifics. We no longer monitor actively during production deployments. They just work, once every 5 months there is a deployment that needs to be escalated to us.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#115

Post by 100011_100001 »

twh270 wrote: post #31
JamesSwift wrote: post #28

Earlier quoted context omitted.

That seems overwhelming to me. Do you feel this is a tenable strategy? Or is the number of deployments becoming an issue?

This is similar to what $myclient is doing. Parent comment doesn't mention whether identification of versions is done manually or whether they just grab master. If the latter, it's probably reasonable. At $myclient, every release to stage and prod requires teams to manually identify each version of each microservice as well as the stories (JIRA tickets) that are being deployed. This is extremely painful, time-consumi…

Our master versions automatically create new images with unique ids and JIRA stories are associated with them. You can see that from JIRA to Gitlab and vice versa.

Having said that there are people that do some manual correlation. Mainly to be able to determine that the fixes actually got in.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#117

Post by Chico75 »

vbsteven wrote: post #8

My usual setup is pretty simple with each service in its own git repository with a Gitlab pipeline: * build code * run tests (unit + integration using database) * build docker image * push to gitlab registry * deploy to staging k8s environment by using a custom image that just templates a .yml and does `kubectl apply` against the staging cluster * optional extra "deploy to production" that works in the same way but i…

The issue with this model of manual deployment to production is that it creates uncertainty about what version was last deployed to, and the team can lose confidence in the deployment process if that doesn't happen regularly

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#120

Post by slifin »

lovedswain wrote: post #14
bhouston wrote: post #4

We just made all the microservices into one big monorepo and we deploy all at the same time. To be honest we tried to avoid the monorepo but it was hellish. Maybe if each microservices was larger and our team was larger but then are they microservices any more?

The biggest difficulty I've experienced is "librification", where some common code ends up in a little library, and soon that library is not so little any more, and not long after starts to look like half of every service. I can maintain discipline when working on small systems alone, but on a team there will always be one lazy person or urgent need which means eventually some shared component gains enough gravity to…

This video was interesting to me

https://youtu.be/pebwHmibla4

Post reply on HN