Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you keep track of releases/deployments of dozens micro-services?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 140 posts

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#61

Post by UK-Al05 »

giantg2 wrote: post #59
UK-Al05 wrote: post #53

Earlier quoted context omitted.

I agree people hop on the microservice bandwagon without really understanding the "philosophy" behind it. Then blame microservices when they struggle.

I actually prefer a monolith compared to the way we do microservices. One or two big apps to own vs ten microservices or apps that require coordinating with others. It's really just a duplication of paperwork and other overhead processes.

We don't require any coordination with ours. It's low cost for us.

We may make the occasional announcement that v5 our api is coming soon which has x features based on feedback from other teams. Then announce when we've released it.

Its a product, and we own it.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#62

Post by giantg2 »

kjeetgill wrote: post #56
giantg2 wrote: post #51

Earlier quoted context omitted.

"If you need to keep track its probably too late. What makes them services is that they should be able to be deployed without a bunch of orchestration of other services." If only you could tell my bosses/architects that. They won't listen to me. Edit: why downvote?

Quick counterpoint: Just because you should be able to release without orchestration doesn't mean you shouldn't be able to watch and track things. You shouldn't have frequent breaking changes but you should still have the tools to manage when you do.

I'm not sure what your counterpoint is in reference to. I didn't see anything about tracking or recovery from breaking changes.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#63

Post by SkyBelow »

killtimeatwork wrote: post #44
BatteryMountain wrote: post #17

Earlier quoted context omitted.

The question is not IF, but WHEN. So ideally you have some kind of monitoring that reports/shows how many services are alive (and where they live in a cluster), how many errors they generate etc. Then based on some thresholds you can take them out of circulation and let them cool down. If certain kinds of errors occurs, or at a certain frequency, the system can notify a site reliability engineer (or equivalent) to ch…

> Production issues are a part of life. Only if you accept them. The alternative is to do very few, rigorously tested releases per year. This way you don't have production issues. That's how industries like banking make sure bank transfers and card payments work and people's money is not randomly lost... It's a shame many other industries just accept their product failing for users as something normal/inevitable.

Then you get the worst of both worlds. You are in an industry where few very well tested releases are needed to meet SLA and customer expectations, but you have enough of the company looking at entirely different industries and wanting to follow their pipeline instead.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#64

Post by giantg2 »

UK-Al05 wrote: post #61
giantg2 wrote: post #59

Earlier quoted context omitted.

I actually prefer a monolith compared to the way we do microservices. One or two big apps to own vs ten microservices or apps that require coordinating with others. It's really just a duplication of paperwork and other overhead processes.

We don't require any coordination with ours. It's low cost for us. We may make the occasional announcement that v5 our api is coming soon which has x features based on feedback from other teams. Then announce when we've released it. Its a product, and we own it.

That sounds nice. My most recent elevation involved coordinating with 5 other services to stop/restart/reprocess items with a couple deploys between the 5. Had to be off-hours too. Sort of normal for us.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#65

Post by taormina »

killtimeatwork wrote: post #44
BatteryMountain wrote: post #17

Earlier quoted context omitted.

The question is not IF, but WHEN. So ideally you have some kind of monitoring that reports/shows how many services are alive (and where they live in a cluster), how many errors they generate etc. Then based on some thresholds you can take them out of circulation and let them cool down. If certain kinds of errors occurs, or at a certain frequency, the system can notify a site reliability engineer (or equivalent) to ch…

> Production issues are a part of life. Only if you accept them. The alternative is to do very few, rigorously tested releases per year. This way you don't have production issues. That's how industries like banking make sure bank transfers and card payments work and people's money is not randomly lost... It's a shame many other industries just accept their product failing for users as something normal/inevitable.

That's adorable. You know that no matter how much testing you do, that something WILL slip through the cracks? Always.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#66

Post by UK-Al05 »

notwedtm wrote: post #21
Cthulhu_ wrote: post #6

Earlier quoted context omitted.

Microservice is a misnomer; it should have a responsibility, but that could be 10 lines of code or 10 million. Anyway, it sounds like you have a distributed monolith. If you cannot maintain and deploy a microservice independently, it should not be a microservice.

I don't build microservices anymore. All of the reasons listed in this thread tend to cause bottlnecks. I aim for domain services. Define your domains, and build a program to service it.

That's microservices. Microservices handle a bounded domain. At least in traditional advice for splitting microservices. The microservice provide an api for that domain.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#67

Post by bhouston »

Cthulhu_ wrote: post #6
bhouston wrote: post #4

We just made all the microservices into one big monorepo and we deploy all at the same time. To be honest we tried to avoid the monorepo but it was hellish. Maybe if each microservices was larger and our team was larger but then are they microservices any more?

Microservice is a misnomer; it should have a responsibility, but that could be 10 lines of code or 10 million. Anyway, it sounds like you have a distributed monolith. If you cannot maintain and deploy a microservice independently, it should not be a microservice.

> Anyway, it sounds like you have a distributed monolith. If you cannot maintain and deploy a microservice independently, it should not be a microservice.

We can maintain and deploy them independently, but it was annoying to try to track which version was deployed where and having to check it out independently, etc.

The overhead was incredibly high. So we plopped them all into a single monorepo as sub projects. We can still update each one individually but we know what is live on the website is what is in the head of that branch.

As someone whose last website was a monolith (Clara.io), we do feel we are getting the benefits of micro services with little of their downsides now. It is like night and day.

It may be we have a lot of micro services for the size of our team - 20+ micro services and a team size of around 12.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#69

Post by UK-Al05 »

giantg2 wrote: post #64
UK-Al05 wrote: post #61

Earlier quoted context omitted.

We don't require any coordination with ours. It's low cost for us. We may make the occasional announcement that v5 our api is coming soon which has x features based on feedback from other teams. Then announce when we've released it. Its a product, and we own it.

That sounds nice. My most recent elevation involved coordinating with 5 other services to stop/restart/reprocess items with a couple deploys between the 5. Had to be off-hours too. Sort of normal for us.

You have all the symptoms of a distributed monolith.

Re: Ask HN: How do you keep track of releases/deployments of dozens micro-services?

#70

Post by edoceo »

giantg2 wrote: post #51
jayd16 wrote: post #42

If you need to keep track its probably too late. What makes them services is that they should be able to be deployed without a bunch of orchestration of other services. You can solve this by having backwards compatible apis. That said, to know what changes would actually break things you'd ideally have a suite of tests.

"If you need to keep track its probably too late. What makes them services is that they should be able to be deployed without a bunch of orchestration of other services." If only you could tell my bosses/architects that. They won't listen to me. Edit: why downvote?

I'm not a downvoter but maybe its cause "they wont listen"

my theory is your presentation is not compelling. Was your CBA clear? What risk/reward metrics did you highlight?

Post reply on HN