Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you keep track of releases/deployments of dozens micro-services?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 140 posts

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#41

Post by jayd16 »

nicoburns wrote: post #23
bhouston wrote: post #4

We just made all the microservices into one big monorepo and we deploy all at the same time. To be honest we tried to avoid the monorepo but it was hellish. Maybe if each microservices was larger and our team was larger but then are they microservices any more?

Perhaps they shouldn't be microservices. What would be the disadvantage if you combined your microservices into a monolith? Or perhaps 4-5 "macroservices"?

In this case I think the disadvantages would be you're locked into one machine type and your blast radius is a bit wider.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#42

Post by jayd16 »

If you need to keep track its probably too late. What makes them services is that they should be able to be deployed without a bunch of orchestration of other services. You can solve this by having backwards compatible apis.

That said, to know what changes would actually break things you'd ideally have a suite of tests.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#43

Post by BurningFrog »

bhouston wrote: post #4

We just made all the microservices into one big monorepo and we deploy all at the same time. To be honest we tried to avoid the monorepo but it was hellish. Maybe if each microservices was larger and our team was larger but then are they microservices any more?

One "wisdom" I hear is that the benefit of microservices is organizational:

One microservice per team, so you cut down on intra-team friction, and the team can manage their own releases.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#44

Post by killtimeatwork »

BatteryMountain wrote: post #17
imafish wrote: post #12

Earlier quoted context omitted.

Do you ever release serious errors into prod?

The question is not IF, but WHEN. So ideally you have some kind of monitoring that reports/shows how many services are alive (and where they live in a cluster), how many errors they generate etc. Then based on some thresholds you can take them out of circulation and let them cool down. If certain kinds of errors occurs, or at a certain frequency, the system can notify a site reliability engineer (or equivalent) to ch…

> Production issues are a part of life.

Only if you accept them. The alternative is to do very few, rigorously tested releases per year. This way you don't have production issues. That's how industries like banking make sure bank transfers and card payments work and people's money is not randomly lost... It's a shame many other industries just accept their product failing for users as something normal/inevitable.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#46

Post by rblatz »

jjice wrote: post #16
monster_group wrote: post #7

I don't keep track. All microservices use continuous deployment pipelines. If you check in code and it passes all the tests, it will make it out to prod some time in the next few hours.

How do updates to a database work through that pipeline? Do migrations run through and rolled back automatically as needed?

We always do it by not pushing breaking changes to the database. It’s extremely freeing. It does require some discipline to go back and cleanup things later, but not worrying about database “versions” is the way to go in my opinion.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#47

Post by lordgilman »

killtimeatwork wrote: post #44
BatteryMountain wrote: post #17

Earlier quoted context omitted.

The question is not IF, but WHEN. So ideally you have some kind of monitoring that reports/shows how many services are alive (and where they live in a cluster), how many errors they generate etc. Then based on some thresholds you can take them out of circulation and let them cool down. If certain kinds of errors occurs, or at a certain frequency, the system can notify a site reliability engineer (or equivalent) to ch…

> Production issues are a part of life. Only if you accept them. The alternative is to do very few, rigorously tested releases per year. This way you don't have production issues. That's how industries like banking make sure bank transfers and card payments work and people's money is not randomly lost... It's a shame many other industries just accept their product failing for users as something normal/inevitable.

I can't say my experience echoes your comment. I'm a former employer of a financial services (billing) company built around a mainframe code base started in the 70s. We probably qualify for the sort of business you had in mind with your comment.

We did four releases a year, across the entire organization (so mainframe and more modern platforms), on Saturday nights/early Sunday mornings. There was plenty of testing but there was still plenty of errors only found on the day of, and rushed to fix in the wee hours or daylight hours of Sunday morning.

The only thing that seemed to correlate with release quality was the overall risk of the release, i.e. the complexity and number of new features written during that quarter.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#48

Post by computershit »

klohto wrote: post #33

Flux with GitOps approach, using Helm charts. All of our of microservices have deployment charts, with frozen image versioning. That way, we can can rollout a whole release knowing they are all compatible with each other and can easily fall back just by using git rollback. CI/CD updates image versions in affected YAMLs on every backend release and Flux keeps staging in sync. When we are happy, we sync to production b…

Have you looked into Jenkins-X at all? I'm at a point where I'm starting to adopt GitOps and I'm torn between Flux and (what I consider) a far more opinionated but pretty elegant solution in JX.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#49

Post by rblatz »

sokoloff wrote: post #20
jjice wrote: post #16

Earlier quoted context omitted.

How do updates to a database work through that pipeline? Do migrations run through and rolled back automatically as needed?

Not in the context of micro-services, but we ran our production DB for years in a “both N and N+1 work” by following a few simple rules which turn out to be not that restrictive in practice. Short version: have DB1 hold the transactional data (data generated while running the system). Have DB2a have the release-bound data (data about and connected to the code itself-settings, prices, whatever). Have DB2a have views o…

This sounds a lot like CQRS
Post reply on HN