When I worked at Atlassian, we had this issue as well, given all the many services that were deployed for products. A few of us left and created Sleuth [1] to solve it for Atlassian and folks like you. Sleuth helps you know what is deployed, its health, and helps with workflow automation. It also tracks the DORA metrics so you know how healthy a service release processes is.
Ask HN: How do you keep track of releases/deployments of dozens micro-services?
91–100 of 140 posts
Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?
#92Earlier quoted context omitted.
I don't build microservices anymore. All of the reasons listed in this thread tend to cause bottlnecks. I aim for domain services. Define your domains, and build a program to service it.
That's microservices. Microservices handle a bounded domain. At least in traditional advice for splitting microservices. The microservice provide an api for that domain.
Doesn't a "service" handle a domain?
Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?
#93I don't keep track. All microservices use continuous deployment pipelines. If you check in code and it passes all the tests, it will make it out to prod some time in the next few hours.
Do you ever release serious errors into prod?
Rarely. All features are gated by feature flags with the capability to dial up the feature gradually and dial down the launch instantly. I can monitor if the feature launch is going as expected by monitoring errors and metrics in the logs.
Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?
#94Since the specific question is "how do you keep track of", my build & deploy script copies a quick one-liner dump of git information (SHA, date, environment, branch, etc.) to a directory on a shared server, as a text file. Later I can go to that server and `cat versions/* | sort` to get a report of what is deployed where/when and so on.
It helps that I have One Deployment Script To Rule Them All (or really, a couple DSTRTA's). When every service has its own special build & deploy script you have to ask nicely and hope people keep up with it. A lot of CI/CD systems force you into that corner because of an implicit assumption that each build & deploy is its own special one-off.
Anyhow, text files rule, at least as an ad-hoc solution.
Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?
#95Flux with GitOps approach, using Helm charts. All of our of microservices have deployment charts, with frozen image versioning. That way, we can can rollout a whole release knowing they are all compatible with each other and can easily fall back just by using git rollback. CI/CD updates image versions in affected YAMLs on every backend release and Flux keeps staging in sync. When we are happy, we sync to production b…
Could you explain more about your "frozen image versioning?"
Was wondering if I just invented the term, or it’s something known :)
Basically a specific semver, no major.minor or just major. Whole version including patch.
Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?
#96Earlier quoted context omitted.
That's microservices. Microservices handle a bounded domain. At least in traditional advice for splitting microservices. The microservice provide an api for that domain.
No, that's SOA. Microservices are almost always described as quite small.
The "Building Microservices" book, which used to be the goto book for microservices suggests on page 31 to page 38 that bounded domains are a good model for microservices.
The main difference between SOA and microservices is dump pipes vs smart pipes. Not service size. As explained in that book.
Most people trying to implement microservices seem to have not read much about them outside of blogs.
Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?
#97Flux with GitOps approach, using Helm charts. All of our of microservices have deployment charts, with frozen image versioning. That way, we can can rollout a whole release knowing they are all compatible with each other and can easily fall back just by using git rollback. CI/CD updates image versions in affected YAMLs on every backend release and Flux keeps staging in sync. When we are happy, we sync to production b…
Have you looked into Jenkins-X at all? I'm at a point where I'm starting to adopt GitOps and I'm torn between Flux and (what I consider) a far more opinionated but pretty elegant solution in JX.
I did, it’s overly complicated for what I need (single team, apply YAMLs in git repo, specific branch, tagging).
I see the industry using mostly Flux and ArgoCD and I really don’t want anything Jenkins related in infra again.
Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?
#98Earlier quoted context omitted.
That's microservices. Microservices handle a bounded domain. At least in traditional advice for splitting microservices. The microservice provide an api for that domain.
Doesn't a "service" handle a domain?
Putting micro in front of service was a bad idea. Lots of people have the wrong idea because of that.
Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?
#99Flux with GitOps approach, using Helm charts. All of our of microservices have deployment charts, with frozen image versioning. That way, we can can rollout a whole release knowing they are all compatible with each other and can easily fall back just by using git rollback. CI/CD updates image versions in affected YAMLs on every backend release and Flux keeps staging in sync. When we are happy, we sync to production b…
Do you have a separate git repo for the deploy config/manifests? Or just force-push your `master` branch to the `staging` and `production` branches to do a deploy (i.e. not keeping full history in the env branches)?
I've seen both advocated for, interested in what the consensus is.
Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?
#100Earlier quoted context omitted.
Time for a job change? Lots of red flags.
I've thought about it. I have no local options and my wife won't move. My remote options are very limited based on the stacks/languages that I have experience in.
So you have few options. Work on more? Pick up another stack. Do some weekend consulting in a new framework. Better yourself. By being stuck in the only 1 job you could possibly work for, you are just delaying the mess that happens when they close down / get bought / you get fired. The best part of programming is you can teach yourself almost anything, especially if you have previous experience to build on.