Live data from Hacker News

Why Segment Went Back to a Monolith

infoq.com

191–200 of 328 posts

Re: Why Segment Went Back to a Monolith

#191
post #50

Earlier quoted context omitted.

Noo! Building teams around software components cements your architecture and prevents most cross-cutting improvements. I'll claim that splitting a well-structured monolith into microservices will always make it less maintanable, but it might be worth it if you need to for some reason like elasticity or failure tolerance. But for the love of god, keep the design open. Don't tie the existence of internal software compo…

It's pretty hard to keep the design open once the whole architecture is bigger than what a single programmer can keep track of. Say, the Linux kernel. The overall architecture is fixed, there's no way around it. At that point, splitting into components that are maintained separately does no harm. AFAIK the Linux kernel is maintained like that already in practice, even if it's a single repo.

I agree with you in such cases, but I'm willing to bet that most codebases don't need to be as big as they are, and that it's better to create an incentive to collaborate and keep the codebase maintanable and small.

Re: Why Segment Went Back to a Monolith

#192
post #105

Are there any case studies where microservices went well? From an end user perspective, Netflix runs in “constantly degraded” mod. From an engineering perspective, they track “number of successful stream starts”, instead of percentage of the time 100% of their services are working. That’s a huge red flag. As a researcher, the monitoring and fault-propagation / modeling work they’ve done to get it to stay up at all is…

Why do you assume they have more fault tolerance scenarios simply because they have more deployable units?

In every service architecture I've worked with in the last few years, you could theoretically run every single "service" within the same physical process -- provided they all shared the same runtime/lang the way a monolith does. I tend to start service architectures using Ruby exclusively with the Eventide toolkit, so this is actually a viable approach for most teams I've worked with. But it's never ultimately made sense. Weighing the pros/cons, a consolidated deployment topology wouldn't add any benefit, and it would actually make it far more difficult for the operations folks, practically speaking.

I've helped put services into production that 1. carry out crucial, "core" business logic reliably and efficiently, 2. can be scaled horizontally without changing the code, 3. never raise exceptions or cause outages, and 4. don't need to be touched for years because new features are more naturally composed around them anyways (i.e. open/closed principle). Practically speaking, it's quite a bit easier for human programmers to reach this degree of precision with a smaller program than with a much larger one. And if you add up a lot of these high quality programs, you get a high quality system. Building a high quality system out of a single program is much, much harder for humans in practice.

I'll use crude terms here for a second: every good service put into production brings a net benefit to the organization relative to the same code having been entangled with an existing pile of code. That may sound like a "No True Scotsman" fallacy, but the definition of "good" in this context is precisely the net benefit being added. If you build and deploy services that have a high degree of quality, you get a corresponding benefit. If you build and deploy programs that _don't_ stand on their own, _don't_ leverage durable messaging, and take other programs down with them when they crash, then you get a rather large mess on your hands. In fact, you never had a service architecture at all; we call this failure mode a "distributed monolith."

I'll acknowledge that most attempts at microservice architectures in the wild don't seem to succeed. Anecdotally, they are particularly prone to failure when their architects don't understand the underlying principles of distributed systems all that well; they neglect important considerations like messaging idempotence, the deleterious effect of synchronous request/response messaging, and the need for deliberate, thoughtful design. In other words, they build systems out of N number of microservices, and get N^2 fault tolerance scenarios, which you adeptly called out for being foolish. They're arguably worse off than if they went with a monolith, but neither would be an architecture I'd personally want to work with.

Re: Why Segment Went Back to a Monolith

#193
post #62

Earlier quoted context omitted.

Noo! Building teams around software components cements your architecture and prevents most cross-cutting improvements. I'll claim that splitting a well-structured monolith into microservices will always make it less maintanable, but it might be worth it if you need to for some reason like elasticity or failure tolerance. But for the love of god, keep the design open. Don't tie the existence of internal software compo…

The opposite of "has a team around it" is "abandoned". Or at least low down on somebody's priority list.

That's generally true, and it's a big problem with microservices, because they need so much upkeep.

But if your code is living as a few hundred or thousand readable lines in the common codebase, that isn't really a problem. The code is there, readable and working, and if anyone needs to change it they can. If it falls out of fashion, it can be deleted.

Re: Why Segment Went Back to a Monolith

#194
post #19

I think that the problem here was that they were fighting against Conway's Law: https://en.wikipedia.org/wiki/Conway%27s_law > Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure. I think microservices work well in organizations that are big enough to have a team per microservice. However if you've just split your monol…

I cannot agree more: I worked at a company where we went from a monolith deployed on IaaS with a couple handful of engineers to Docker containers deployed on ECS with over 200 engineers. The main reason we did it was because Docker+ECS was cheaper than a bunch of EC2 instances and you can't effectively use 200+ engineers with a single monolith.

After 2 years we had over 450 microservices while keeping our AWS bill flat or slightly decreasing.

Re: Why Segment Went Back to a Monolith

#195

Earlier quoted context omitted.

Noo! Building teams around software components cements your architecture and prevents most cross-cutting improvements. I'll claim that splitting a well-structured monolith into microservices will always make it less maintanable, but it might be worth it if you need to for some reason like elasticity or failure tolerance. But for the love of god, keep the design open. Don't tie the existence of internal software compo…

What is your alternative? Tying "the existence of internal software components to people's livelihoods" across the expanse of the entire codebase is the only remotely effective approach I've seen to scaling the SDLC at scale.

Think of all the open source libs. Generally speaking, anyone can contribute to any part of the project.

That's not to say that some people are better than others at certain parts of the codebase, but you don't want people fighting to keep old cruft in because it's on their job title (figuratively speaking).

You can organize around customers, use-cases, platforms, concerns or other things. Some might naturally map 1-1 to software components, but the software component should not be the "reason d'aitre" for a team, rather the customer experience or something else which can transcend multiple interations of the software.

Re: Why Segment Went Back to a Monolith

#196

Earlier quoted context omitted.

There is nothing wrong with most developers working on and communicating about the entire code base. Having teams work in silos is not a benefit. You're touting as a benefit what is one of microservices' gravest issues - teams stop communicating beyond the surface level of their respective APIs.

Have you tried coordinating entire teams to work on a shared codebase? Honestly, I have never been in an organization so large that this became a necessity (if you solve tens of different problems, that would require almost thousands of developers). But coordinating single developers without an API is hard enough already, I can only assume for teams its nearly impossible.

Define “codebase”? You can have multiple services, user facing apps or modules inside a single repository, but if there are no boundaries coordination will be difficult of course.

Re: Why Segment Went Back to a Monolith

#197
post #19

I think that the problem here was that they were fighting against Conway's Law: https://en.wikipedia.org/wiki/Conway%27s_law > Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure. I think microservices work well in organizations that are big enough to have a team per microservice. However if you've just split your monol…

I don't agree with your premise that development structure == deployment structure. There are plenty of good ways of splitting up development of a monolith without the huge devops headache of deploying microservices.

Re: Why Segment Went Back to a Monolith

#198
post #95

Earlier quoted context omitted.

Mmmmyes and no. Depending on the size of your project, that may not be the case. I've had to work with two titans of monoliths, maintained by relatively small teams(anywhere between 2 and 6-7 people for several million lines of code). At some point managing a codebase this big within a single project becomes a huge burden, for both developers and even more so for those who develop and do code-reviews(first hand exper…

Good monoliths are highly modularized. But it's a whole different thing to package up a module as a separately deployable unit for external "public" use (external to your app, that is, not your company). I'm just curious to know, when you said "the easiest and most sensible thing to do is chunk out large parts of the project and put them aside as a microservice" ... were these chunks separately deployable units for e…

Oh, all of those were heavily modularized to begin with. But that wasn't enough to keep them manageable. So at the end what we did is figure out which are the core components between the different modules, isolate what they did and put them aside in a smaller microservices, which were easier to track, maintain and monitor. What was once the monolith is now arguably just an interface/API for all the heavy lifting which is done by microservices. Again, my point is that all this must be done depending on the scale and complexity of your application. If you are going to make an authentication microservice for an application that has 50,000 users which simply fetches a username and compares a hash in a database, obviously you are doing it wrong. I am talking about applications which in the simplest of times operated on 24 different databases located in completely different geographical locations(the case of my first such monolith). Some of those databases used different engines. And due to the nature of the infrastructure and the requirements we couldn't simply ditch everything and start over from scratch. So splitting everything into microservices was the only option. And this is something I was working on back in 2012 iirc so back when microservices were considered witchcraft by most people. And yes, I'm talking about several million lines of code and 2 developers - my inexperienced out of uni ass, and an utterly conservative dev twice my age. Took us around 6 months but the project was extremely successful.

There is this trend in technology - every few years everyone changes their minds about everything:

* 2012 - sql is the best.

* 2016 - sql sucks, nosql is the future

* 2020 - nosql suck, sql is the best.

* 2024 - {fill in the blank}.

The same thing is happening with microservices. But in addition docker, kubernetes and recently unikernels have joined the party. The concept is the same though.

What I am trying to say is that either of those can be good or bad in different scenarios. It's a question of picking the most appropriate one for the situation.

Re: Why Segment Went Back to a Monolith

#199

They went to 50+ services in a few months, were applying the same policies across all services. It sounds like they didn't plan well enough and jumped straight into it, without any good DevOps or infrastructure mindset. It was a disaster waiting to happen. This shouldn't be an article that people read and say "Oh Im never using Microservices". This should be an article people read and say WOW that is exactly the righ…

[deleted]

Re: Why Segment Went Back to a Monolith

#200

I see quite a few people defending microservices; the org is the problem, they must not have written the software correctly, etc. Most org structures are not great. Most software is not great. If you expect the exception to be the rule you're setting yourself up for a career full of disappointment. Microservices are a modern re-branding of service-oriented architecture, but 'microservices' sounds cuter and less like…

> split along domain lines with minimum inter-service dependencies. Exactly, and done right that quite often means big 'microservices'. All too often I see the 'functional programming disease' where the aim is to deconstruct to the smallest possible reusable functions ( 'micro' services right? ), often prematurely, creating high levels of compositional complexity and with zero tools to help you understand how the act…

Completely agree. The micro thing is rooted in a desire for simplicity, which is laudable from a theoretical perspective. Small is simple and simple is nice to work with. But in any production software it's usually a pipe dream.

Kind of the whole point of software is to take a bunch of complexity and make it simpler for the end user. The fact that the software visually looks like shit or is difficult to work with because it's difficult to reason about is a rather unfortunate side-effect, but usually has no bearing on actual business success.

There are lots of things we should try to do to reduce software complexity and make it easier and safer to work with and change. But trying to force simplicity by way of size usually has the opposite effect.

Post reply on HN