Why Segment Went Back to a Monolith

If the question "are there any case studies where microservices went well" is a valid question, then so must be, "are there any case studies where monolithic architecture went well".

My point being that if and only if we have a track record as an entire field of making decisions based on case studies, AND if case studies have a track record of being objective rather than proffered as a result of a marketing agenda, then the question is ultimately legitimate.

There are more shops by total count that fail with monolithic architecture. That's inevitable just based on the infinitesimal number of projects executed as service architectures rather than monoliths at large in the wild. But still, we carry on with monolithic architectural style as if its outcomes were assured.

It's far easier for the vast majority of developers to build a monolith because it allows development to proceed without having to have any knowledge of or practice with the tricks and traps of distributed systems - an entire body of knowledge that a developer might never get meaningful and practical exposure to for an entire career.

The trouble starts when microservices are attempted by developers who can't imagine that there are entire bodies of software development knowledge that developers aren't presently in possession of.

Microservices is just, as Adrian Cockroft used to say, "Service-Oriented Architecture with bounded contexts".

Web development is absolutely not a preparatory course in service oriented architecture. But the vast majority of web developers who attempt to take on SOA while simultaneously presuming an omniscience in all things software development due to their experiences only with monolithic web development will often fail to build a SOA. They usually end up with something that isn't quite SOA and isn't quite a monolith. And that's where the failures largely come from.

I work exclusively in microservices and SOA, and have since 2015. Before that, I worked principally as a web app developer, and did some work off-and-on in SOA implementations. And before that I spent years becoming oriented to the architecture. I don't make the mistakes that web developers typically do when they presume that web development knowledge is a sufficient prerequisite for working in SOA.

So, it's not a question of whether an architectural style works or doesn't. The majority of failures in microservices can be attributed to ignorance and to the narcissism that is permissive of it.

So, I would ask this question instead: Are there any cases where developer over-confidence and over-simplification went well?

These qualities don't tend to serve any architectural style well.

I've never heard of a well-designed SOA not going well. Every single case of microservice project remediation that I've participated in had as the most significant contributing factor an utter disregard for the body of knowledge that the microservices architectural style is built upon.

There are a lot of things that developers can get away with when doing the kinds of tinkering and wandering that typifies typical web development work. But those things don't work once we cross the line into SOA. And unfortunately, the incessant chasing after trivial resumé candy hasn't prepared the average developer for the rigorous mindset needed for SOA work.

As "microservices" became to next fad for perennial fad chasers of the software development world, they finally encountered a kind of work that they could not get away with by faking it. And so, we see a lot of failures. But the vast majority of the failures are personal failures and character failures, rather than failures of an architectural style.

The fat part of the developer bell curve was simply overreaching when it presumed to try to get away with building service architectures with the same level of disinterest in architecture and process that we can get away with in typical web development. Like a kid with copious experience building kites presuming to strap themselves to a hang glider and just "going for it". The outcomes are mostly predictable.

In the end, if a little time is invested in learning the fundamentals that have so far been eschewed for the sake of expediency, anyone can succeed with SOA and microservices. It's not that the realities of the architectural style are unlearnable, but it can't be arrived at by the level of tinkering and wandering that we can just get away with in monolithic web development.

I see a lot of places that seem to either think that:

1. Microservices will let them ship things faster or

2. It's microservices everywhere or nothing

Microservices might let you ship faster if you are really good at deciding where to draw the lines between services and really good at managing multiple deployment pipelines and all the infra - that's a pretty tough ask.

Also, if you have a monolith it's perfectly fine to pull out one or two parts that need to scale much more efficiently and leave most of your codebase in the monolith, but a lot of times I see companies think once you have created one microservice the monolith is now the worst thing possible and it needs to be broken up entirely.

My general rules for this are to always start in a monolith and break things out as they start to fail or break other parts of the codebase, and don't go all in just because you now have one microservice that works well by itself

Indeed my loudest prescription to people doing service-oriented architectures of any kind is to simplify these arrows.

The common mistakes that I see are for two services to share read access to a common database, or to discover each other and send RPCs to each other. Both really dangerous for exactly this reason! The common database obscures how the two communicate with each other, and invariably everything connected to a database becomes one service -- call it a "mini-lith" if the overlapping sets created by databases do not cover the whole architecture. The problem is the preponderance of implicit arrows; when I reason about what it means to make this datetime nullable so that I can store such-and-so, I need to consider whether everybody who can read that datetime will be prepared for its nullability.

RPCs and APIs are the same way. I add a contract about what I am outputting and then everybody needs to know about my contracts and I must commit to them or else modify all of my consumers. So because the arrows are bi-directional everything just becomes one monolith again.

Instead, I recommend message brokers -- all that pubsub stuff. A given service tells all the other services simultaneously "this happened," and it is their responsibility in their codebase to listen for that event and then say "okay, then this must happen." Publishing a new version of the event is done by just emitting both the old and the new version of the event and perhaps having a shared standard for deprecation across the codebase so that you get deprecation warnings in your prod logs.

Every service has its own database and they generally only communicate to each other through these broadcasts, makes the arrows into the "stuff you do see".

I think there's an element of truth to the engineers' claims. Working on different code bases means there are a lot of things you would otherwise need to talk about that now you don't. It's very much the case that you still need your interfaces to be clear (in fact, clearer!) but those discussions can be somewhat isolated, so more work can proceed asynchronously. Just how isolated depends on how exact (and correct) the specifications are, which is a question of trading up-front work against interruption.

> I think microservices work well in organizations that are big enough to have a team per microservice.

Presumably by definition we’re talking about a few hundred lines of code, or a couple of weeks development time here at most. What does this team do all day otherwise?