Why our team cancelled our move to microservices • Hacker News Board

Re: Why our team cancelled our move to microservices

#211

Post by winternett » Wed, Jan 26, 2022, 4:28 AM UTC

throwaway894345 wrote: post #199

winternett wrote: post #180

Earlier quoted context omitted.

Many "Monoliths" can be run on something as simple as a desktop emulator. They don't generally rely on cloud either. Microservices are "distributed" across a cloud host platform because they are each updated and maintained by different teams. My use of the term "distributed means that if AWS East has your DB instance and your web server is stored in an entirely different region, you app goes down anyway, but your fro…

> Many "Monoliths" can be run on something as simple as a desktop emulator. I'm not sure what a "desktop emulator" is, but a lot of micro services can run as native processes or in a VM. There's nothing about micro services that fundamentally restrict where they run--they're just application processes at the end of the day. > my use of the term "distributed means that if AWS East has your DB instance and your web ser…

FTR... A desktop emulator was a reference to something like WAMP or XAMPP... On which a monolith can be run, developed upon, and even tested entirely independent of any host or VM infrastructure.

I'm not back pedaling, you're reading too closely and judging instead of looking at the discussion from an objective standpoint and simply seeking clarity and working towards truth. Here's what I posted in this same thread even before my post in this branch -

+++

A size 20 shoe is better for a large foot... But not better for a size 15 or size 10 foot. Saying Microservices are better is the same as me saying "a size 20 shoe is better than any other shoe"... for everyone.

It's not a viable statement in any use case, except for people with size 20 feet.

The business need is what determines the solution necessary.

Re: Why our team cancelled our move to microservices

#212

Post by kgeist » Wed, Jan 26, 2022, 4:46 AM UTC

nchi3 wrote: post #148

kgeist wrote: post #120

We have a hybrid model: modular monolith + microservices. It has worked well so far. The core of the product is found in the monolith. We use bounded contexts ("modular monolith") with strictly separated concerns. There are no immediate plans to split the core into microservices (unless absolutely necessary) because the logic between modules is too intertwined and coupled. Splitting the core into microservices would…

> We use bounded contexts ("modular monolith") with strictly separated concerns. >because the logic between modules is too intertwined and coupled. That doesn't sound very modular? If your bounded contexts are intertwined, I don't think they can be considered bounded contexts. A modular monolith would only communicate between contexts through well-defined and non-leaky APIs, and that's the opposite of intertwined.

They only communicate through well-defined APIs, and there's a rule that cross-context API calls can only happen in the anti-corruption layer (we use a tool to check it at build time).

What I meant by intertwined (maybe a wrong word, I'm not a native speaker):

1) there's a lot of data/logic dependency between the contexts (i.e. a context in its operation depends on N other contexts), although we at least disallow circular references; it's unfortunately dictated by business rules and I'd like to see contexts to be more isolated and self-contained. Some can say that if a change in the requirements requires to change many contexts at once, maybe it's one fat context after all - and they may be right, but we enjoy the current modularization effort, one big fat module would be far less manageable for us.

2) there're occurrences of temporal coupling; there are synchronous operations that spawn several contexts, with a lot of data flowing back and forth

Now, it's easier to manage it in a monolith, in the same process, because:

1) there are no network trips back and forth in case of complex operations with a lot of data

2) no retry logic in case of network connectivity issues

3) DB connections/locks and other in-memory structures can be reused

4) same codebase, so easier to reason about

Microservices require more care and more complex solutions:

1) distributed transactions are hard

2) eventual consistency is hard

3) the idiom "DB per microservice" makes managing the infrastructure harder

4) deployment is harder (if you have changes in several related contexts, there's only 1 deployment in the monolith as opposed to N deployments of microservices)

5) you have to manage different codebases/repos, can't see the whole picture

6) you have to defend against network connectivity issues, microservice unavailability etc.

7) debugging is harder, you can't just step into another microservice like you do with in-memory modules

8) new devs need to be taught all that

The list can go on and on. So we don't try to make all our modules/contexts into microservices just because we like microservices, we have to substantiate a move to a microservice with proof that it will make development/scalability easier for us, and that the advantages outweigh the disadvantages.

Re: Why our team cancelled our move to microservices

#213

Post by hackerfromthefu » Wed, Jan 26, 2022, 5:09 AM UTC

jedberg wrote: post #114

wpietri wrote: post #104

Earlier quoted context omitted.

One of the questions I like asking developer pals is what ratio their company has between engineers and services/deployable units. Anybody reading this care to share? For me, that number says a lot more about the day-to-day life of devs than the microservices vs monolith label does.

I don't remember the hard numbers, but on average each service at Netflix was maintained by about 4 people but there were outliers in both directions. Sometimes there were four or five services maintained by one person, and sometimes there was one service backed by a team of 25+. The other important number was that about 25% of engineering was dedicated to building the tools to manage the microservices. We didn't wor…

For anyone with a monolith considering microservices, that 25% of total is 33% more work than the monolith.

But if 33% more work isn't enough, we can things in languages with undefined behaviour like Javascript, and also use lots of different languages in the services so out support gets more complex!

Jokes aside, the saving grace is t25% sounds like empirical observation so it probably an all up figure so probably includes the large amounts of bikeshedding that often accompanies a microservices implementation.

Re: Why our team cancelled our move to microservices

#214

Post by kgeist » Wed, Jan 26, 2022, 5:42 AM UTC

noogle wrote: post #170

ARandumGuy wrote: post #78

Earlier quoted context omitted.

There isn't really a solution to version bloat aside from good processes and general diligence. There's no easy way to handle that sort of thing, unfortunately. However, I do think it can be easier to deal with for internal services then for something like Java. When the number of users is in the dozens rather then the millions, it's a lot easier to make sure everyone gets moved over to the new version.

That's what makes the whole idea of microservices seems weird to me. If a functionality has merit on its own (e.g. an authentication service), then it will naturally fall outside of the main application. If a service is tightly coupled to other parts of the app, then microservices seems like intentionally hindering yourself: the coupling remains (as evident by the need for backward compatibility), but now we it's har…

I agree that tightly coupled modules should live inside the same deployment, but:

>it's harder to keep everything aligned due to the extra separation (e.g. different code bases, multiple databases, no static validation of remote interfaces etc.)

It's solvable with appropriate tooling. I.e. you can store API definitions in a separate repository and make the services or CI/CD check API usage is valid at build time.

>tightly coordinated machine

What do you mean by that? For example, we have 10 teams all developing different features in parallel, with a tight release schedule. If there was tight coordination for every change, we'd degrade to a waterfall on the scale of the whole organization. Major API changes are discussed in advance during P/I planning; for minor changes, it's a matter of simply notifying other teams "hey, add this to your backlog, please" (we enforce backward compatibility for zero downtime anyway, so it's not urgent)

Re: Why our team cancelled our move to microservices

#215

Post by yakshaving_jgt » Wed, Jan 26, 2022, 9:21 AM UTC

fdgsdfogijq wrote: post #6

It took me a while to accept that microservices are better. Not in every case, but in a surprising number of cases. They really shine when combined with serverless computing. Clear seperation of code by a networking call is the next logical step in the encapsulation principle of object oriented programming. We hide the implementation details and only expose an interface, which creates seperation and forces us to stop…

Separating services with network requests doesn’t stop you having “spaghetti logic”. All it does instead is add more spaghetti and put it in different bowls.

Re: Why our team cancelled our move to microservices

#216

Post by tentacleuno » Wed, Jan 26, 2022, 10:31 AM UTC

phphphphp wrote: post #193

tentacleuno wrote: post #186

> Because we couldn’t isolate any of our services properly, this was going to mean that we would be left with a significant amount of duplication. For example, we identified one particularly complicated and essential piece of business logic that would have to be copy-pasted and maintained across 4 of the planned microservices. Wouldn't this piece of business logic be best placed in an import-able module? Then, that m…

I have a very strong objection to this line of thinking. Effective use of microservices depends upon a strong, meaningful boundary between the services and that boundary should be business driven, not code driven. As soon as you start dealing in packages of code[1], there’s no longer a meaningful boundary between your services, instead the boundary is completely arbitrary and each service becomes a microservice in na…

> As soon as you start dealing in packages of code[1], there’s no longer a meaningful boundary between your services, instead the boundary is completely arbitrary and each service becomes a microservice in name only.

While this sounds very radical (to me at least), I mostly understand how you've come to this conclusion. Obviously "just one package" is going to lead to further complexities down the line, and perhaps many more packages than that.

Perhaps a dedicated microservice for this piece of business logic would be better, as you suggested.

Re: Why our team cancelled our move to microservices

#217

Post by biorach » Wed, Jan 26, 2022, 10:42 AM UTC

serial_dev wrote: post #102

winternett wrote: post #7

The term "Monolith" was devised by people who wanted to brand microservices as newer and superior. It's almost as if in order to succeed these days you need to discredit and disparage your competition rather than simply having a better product, and that's why I don't buy into buzz words at all. If it's not broken, don't fix it... Microservices are relatively new and unproven. The way the world has rushed to dive into…

Microservices make sense in some scenarios. I work at a large retail company with who knows how many developers. We have different teams for payment, promotions, product search, account, shipping and more. All of them working on a single codebase with coordinated deployments would be a nightmare. Previously, I joined a startup (previous coworkers of mine), a developer and a business guy. The developer "drank the micr…

Yes! This really sums it up - they have their place, but it takes experience and good judgement to know where they are appropriate, and how to divide up concerns.

Re: Why our team cancelled our move to microservices

#218

Post by bluehatbrit » Wed, Jan 26, 2022, 1:30 PM UTC

wpietri wrote: post #104

jedberg wrote: post #61

> We have approximately 12 developers spread across 2 feature teams and a support team. If I were consulting for this company, I would have told them to stop right there, microservices are probably not for them. Unless you build from the start for microservices on something like AWS lambda, doing with such a small team would be really hard. And as they eventually discovered, a lot of unnecessary overhead for such a s…

One of the questions I like asking developer pals is what ratio their company has between engineers and services/deployable units. Anybody reading this care to share? For me, that number says a lot more about the day-to-day life of devs than the microservices vs monolith label does.

At my last place, a team of about 4 would usually handle maybe 2 services, possibly a 3rd if it was some kind of light weight shim. Larger teams towards about 8 might have 4-5 components. There were some outliers as ever in an org with 100's of engineers, typically teams who had more components struggled to deliver their roadmaps. They'd also often end up with huge blocks of work when a bit of technical debt cropped up as it was complicated by having more services.

We eventually got to the point where we started re-definining everything in terms of "business processes" (ie: creating a basket, paying for an order, making a complaint, etc). Teams were moving towards talking about processes rather than services, how we chose to organise the code and deploy it was starting to become more about practicalities. Such as: this collection of endpoints services an internal tool and scales differently to this other collection which service customer facing clients.

I left not long ago but my team of around 7 engineers had 2 primary service where 90% of the work happened. The other 10% of the work was on a few edge services, typically small serverless functions, which were mostly set and forget. It felt like we could move at a good pace with this setup. My new job is even more monolithic leaning and it's even faster to get work done and build internal tools.

Re: Why our team cancelled our move to microservices

#219

Post by nurettin » Wed, Jan 26, 2022, 1:31 PM UTC

linuxdude314 wrote: post #5

nurettin wrote: post #2

> A benefit of microservices is that each team can be responsible for releasing their services independently and without coordination with other teams. Sounds almost sarcastic. How do you deliver API changes without alerting other teams?

Do you not version APIs you design? When engineering an API meant for consumption by disparate services it’s imperative to provide back words compatibility. This is pretty basic stuff anyone designing a serious API should be taking into account.

Yes, and I have to consult with other teams when versioning, I don't just go "welp here's the next version I created and deployed without asking anyone because that's what the IBM microservices manifesto (2006) suggested and some random guy on HN insinuated I'm causing undue friction if I work with you guys"

Re: Why our team cancelled our move to microservices

#220

Post by serial_dev » Wed, Jan 26, 2022, 1:47 PM UTC

withinboredom wrote: post #171

serial_dev wrote: post #102

Earlier quoted context omitted.

Microservices make sense in some scenarios. I work at a large retail company with who knows how many developers. We have different teams for payment, promotions, product search, account, shipping and more. All of them working on a single codebase with coordinated deployments would be a nightmare. Previously, I joined a startup (previous coworkers of mine), a developer and a business guy. The developer "drank the micr…

There’s really not much coordination with a monolithic deployment. Merge your commit and get in line. Monitoring tools will ping you on slack if there’s any issues. It’s probably easier than micro services because all tooling and builds are centralized.

> Merge your commit and get in line.

Much easier said than done, if there are 50+ devs working on the same service (monolith). Decisions are made much slower, improvements need to go through long discussions, deployments are much more risky, onboarding is a hassle, and so on. Sure, monoliths work very good for small teams (it's hard to define what "small" is), but I'd say that around 15-30 developers you can think about splitting up.