Live data from Hacker News

Ask HN: How do you keep track of releases/deployments of dozens micro-services?

news.ycombinator.com

31–40 of 140 posts

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#31

My team is responsible for about 200 microservices being deployed, some of them have 10 or more pods. We don't do continuous delivery. Instead it's done by 5 different groups deploying once every week or two. Our production deployment jobs are in Jenkins and isolated. It's easy to check what was deployed when. We also have a script written that can run an environment report to see what versions and which microservice…

That seems overwhelming to me. Do you feel this is a tenable strategy? Or is the number of deployments becoming an issue?

This is similar to what $myclient is doing.

Parent comment doesn't mention whether identification of versions is done manually or whether they just grab master. If the latter, it's probably reasonable. At $myclient, every release to stage and prod requires teams to manually identify each version of each microservice as well as the stories (JIRA tickets) that are being deployed. This is extremely painful, time-consuming, and error-prone. Avoid at all cost; as the number of services grows, the pain/time/error cost appears to increase geometrically.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#32
Two ways I've seen it done reasonably well.

The somewhat more modern way with Kubernetes deployments is the Helm "chart of charts" pattern, where your system level deployment is a single Helm chart that does nothing but pull in other charts, specifying the required semantic version of each sub-chart in the values.yaml file.

The older, but also much more flexible way I've seen it done is through something a local system architect developed a while back that he called a "metamodule." This was back when Apache Ivy was a preferred means of dependency management when Apache Ant was still a popular build tool and microservices were being deployed as Java OSGi components. Ivy defines a coordinate to uniquely identify a software dependency by organization, module, and revision. So a metamodule was just a module, but like the chart of charts, it doesn't define an actual software component, but rather a top-level grouping of other modules. Apache Ivy is significantly more flexible than Helm, however, allowing you to define version ranges, custom conflict managers, and even multiple dependencies that globally conflict but can be locally reconciled as long as the respective downstreams don't actually interact with each other.

Be aware both of these systems were for defense and intelligence applications. Personally, I would just recommend trunk based development and fail fast in production for most consumer applications, but for things that are safety or mission critical, you can't do that and may have very stringent pre-release testing and demonstration requirements and formal customer acceptance before you can release anything at all into ops, in which case you need the more complicated dependency management schemes to be able to use microservices.

Arguably, in this case, the simplest thing to do from the developer's perspective is don't use microservices and do everything as a monorepo instead, but government and other enterprise applications usually don't want to operate this way because of being burned so much in the past by single-vendor solutions. It's not totally impossible to have a monorepo with multiple vendors, but it's certainly a lot harder when they tend to all want to keep secrets from each other and have locally incompatible standards and practices and no direct authority over each other.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#33
Flux with GitOps approach, using Helm charts.

All of our of microservices have deployment charts, with frozen image versioning. That way, we can can rollout a whole release knowing they are all compatible with each other and can easily fall back just by using git rollback.

CI/CD updates image versions in affected YAMLs on every backend release and Flux keeps staging in sync. When we are happy, we sync to production branch, Flux syncs and it's done.

If we spot an issue that we didn't see in staging, we either release a hotfix or rollback.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#34
post #16

Earlier quoted context omitted.

How do updates to a database work through that pipeline? Do migrations run through and rolled back automatically as needed?

Not in the context of micro-services, but we ran our production DB for years in a “both N and N+1 work” by following a few simple rules which turn out to be not that restrictive in practice. Short version: have DB1 hold the transactional data (data generated while running the system). Have DB2a have the release-bound data (data about and connected to the code itself-settings, prices, whatever). Have DB2a have views o…

Ran similar setup to this and worked pretty well!

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#35

I'm curious to see how micro service management evolves over time and learn whether or not it will become viable for small companies. Hopefully one day its as cheap as writing a function is. As it stands, with what I've seen and heard about microservices, I'd say the best way to deal with micro service anything is to use a monolith 90% of the time and for the rest of the time make sure your micro service could stand…

Right now I don't think microservice management is 'viable' even at larger companies. The custom deployment scripts & yaml to manage building, package/artifact repository, versioning, and deployment tends to be Too Damn Big, at least at the shops I've seen.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#36

Earlier quoted context omitted.

The biggest difficulty I've experienced is "librification", where some common code ends up in a little library, and soon that library is not so little any more, and not long after starts to look like half of every service. I can maintain discipline when working on small systems alone, but on a team there will always be one lazy person or urgent need which means eventually some shared component gains enough gravity to…

I don't get the first paragraph you are saying. This lazy person puts some random code into a shared component, or... ? Wouldn't this urgent need mean that they put this code into the microservice that needs this urgent update as opposed to going through the effort to make it available for everyone to use?

[deleted]

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#37

My team is responsible for about 200 microservices being deployed, some of them have 10 or more pods. We don't do continuous delivery. Instead it's done by 5 different groups deploying once every week or two. Our production deployment jobs are in Jenkins and isolated. It's easy to check what was deployed when. We also have a script written that can run an environment report to see what versions and which microservice…

Any plans for showing user's birthday on the settings page?

Sorry, I'm just kidding but that's the only thing I could think off when I heard the number 200!

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#38
post #33

Flux with GitOps approach, using Helm charts. All of our of microservices have deployment charts, with frozen image versioning. That way, we can can rollout a whole release knowing they are all compatible with each other and can easily fall back just by using git rollback. CI/CD updates image versions in affected YAMLs on every backend release and Flux keeps staging in sync. When we are happy, we sync to production b…

Could you explain more about your "frozen image versioning?"

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#39
post #16

Earlier quoted context omitted.

How do updates to a database work through that pipeline? Do migrations run through and rolled back automatically as needed?

Not in the context of micro-services, but we ran our production DB for years in a “both N and N+1 work” by following a few simple rules which turn out to be not that restrictive in practice. Short version: have DB1 hold the transactional data (data generated while running the system). Have DB2a have the release-bound data (data about and connected to the code itself-settings, prices, whatever). Have DB2a have views o…

This sounds interesting! is there any more detailed write up that you link me to? Thanks!

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#40
post #16

Earlier quoted context omitted.

How do updates to a database work through that pipeline? Do migrations run through and rolled back automatically as needed?

Not in the context of micro-services, but we ran our production DB for years in a “both N and N+1 work” by following a few simple rules which turn out to be not that restrictive in practice. Short version: have DB1 hold the transactional data (data generated while running the system). Have DB2a have the release-bound data (data about and connected to the code itself-settings, prices, whatever). Have DB2a have views o…

Whoa I love this idea. I usually refer to it as "transactional" (always growing, represents the daily activity, read and write heavy) vs "lookup" data (almost exclusively read, not changed often). But still store them in the same database.

What ends up happening is we end up separating the two more with "cache this one, not that one" rather than two different databases.

I will explore this idea on my next greenfield project, whenever that happens.

Post reply on HN