Why Segment Went Back to a Monolith • Hacker News Board

Re: Why Segment Went Back to a Monolith

#121

Post by musingsole » Wed, Apr 29, 2020, 2:27 PM UTC

hedora wrote: post #105

Are there any case studies where microservices went well? From an end user perspective, Netflix runs in “constantly degraded” mod. From an engineering perspective, they track “number of successful stream starts”, instead of percentage of the time 100% of their services are working. That’s a huge red flag. As a researcher, the monitoring and fault-propagation / modeling work they’ve done to get it to stay up at all is…

> Are there any case studies where microservices went well?

You answered your own question with Netflix. While you're right that it's not clear Netflix would've needed to develop their chaos monkey tooling and the like, it's not clear at all at that an equivalent system is possible as a monolith. Even if a monolithic Netflix system were technically possible, it's not clear if a monolithic system would be organizationally feasible either (Conway's Law)

Re: Why Segment Went Back to a Monolith

#122

Post by brootstrap » Wed, Apr 29, 2020, 2:27 PM UTC

"voices of experience pointing out that most decisions are made based on the best information available at the time."

funny in my experience with digital and particularly larger corporate jabronis. The people who makes decisions are fucking McKinsey consultants who know nothing about the actual project, are only contracted for 6 months, then they are gone. Rinse and repeat, maybe one out of every 3 or 4 attempts somebody actually gets it right and the project doesnt completely fail.

Re: Why Segment Went Back to a Monolith

#123

Post by vorpalhex » Wed, Apr 29, 2020, 2:32 PM UTC

hedora wrote: post #105

Are there any case studies where microservices went well? From an end user perspective, Netflix runs in “constantly degraded” mod. From an engineering perspective, they track “number of successful stream starts”, instead of percentage of the time 100% of their services are working. That’s a huge red flag. As a researcher, the monitoring and fault-propagation / modeling work they’ve done to get it to stay up at all is…

Microservices working well is entirely about the team, and my current group of teams works very well with the monolith pattern. A big part of this is because (for business reasons, not technical ones) they frequently trade ownership of parts - so if a service isn't well constrained it will be very quickly. We also have mature DevOps practices and engineers handle significant parts of DevOps themselves instead of just kicking the can.

But the reason I say this is about the team is because I've seen a well built, well groomed service be passed off to an outside team and immediately turned into a disaster. Same service, same business case, just a team without the DevOps savvy and willingness to follow the patterns.

Netflix does some.. unusual things with microservices, mostly with how they treat version rollovers. It's not bad or good, but it looks different from how many other shops handle the same problem and it means asking the question "is everything working?" is extremely difficult but asking the question "how many people are able to start watching?" is pretty easy.

Re: Why Segment Went Back to a Monolith

#124

Post by joshdick » Wed, Apr 29, 2020, 2:32 PM UTC

BareNakedCoder wrote: post #103

axegon_ wrote: post #95

Earlier quoted context omitted.

Mmmmyes and no. Depending on the size of your project, that may not be the case. I've had to work with two titans of monoliths, maintained by relatively small teams(anywhere between 2 and 6-7 people for several million lines of code). At some point managing a codebase this big within a single project becomes a huge burden, for both developers and even more so for those who develop and do code-reviews(first hand exper…

Good monoliths are highly modularized. But it's a whole different thing to package up a module as a separately deployable unit for external "public" use (external to your app, that is, not your company). I'm just curious to know, when you said "the easiest and most sensible thing to do is chunk out large parts of the project and put them aside as a microservice" ... were these chunks separately deployable units for e…

You're right, and I think you've highlighted what makes a good monolith so hard to build and maintain.

You need to be disciplined to keep a monolith highly modularized. For microservices, in contrast, their architecture encourages modularization.

Re: Why Segment Went Back to a Monolith

#125

Post by musingsole » Wed, Apr 29, 2020, 2:39 PM UTC

gilbetron wrote: post #97

lifeisstillgood wrote: post #10

I have this thing about micro-services/complexity in that it follows Conway's Law - the architecture follows the organisational structure. If you push authority and decision making and responsibility for a service to a (2 pizza) team then guess what, microservices work really well. If you have vast monolithic centralised production operations teams, and no way in hell is their C-Exec going to assign two of them to lo…

Conway's Laws isn't a law, it's just an interesting thought experiment. Organization and architecture bidirectionally effect each other, but not directly, and not completely. I hate how current discourse invokes these different "Laws" as if they are physical properties of the universe. I've worked at places with a strong, hierarchical organization that created a wonderful set of "micro" services, and I've worked at p…

Conway's Law is a physical law in the same sense as Murphy's Law.

It's also obviously true. The organization builds the architecture. The architecture either helps or hinders the organization. The organization builds a new architecture. There's no indirect connection here. If you've seen hierarchical organizations implement microservices, it's because that organization's complement was a microservices architecture. And likewise for a chaotic organization.

--well, sidetrack: Aren't strongly hierarchical organizations the best suited for microservices? With all the strongly divided responsibilities and whatnot?

Re: Why Segment Went Back to a Monolith

#126

Post by marcosdumay » Wed, Apr 29, 2020, 2:42 PM UTC

hedora wrote: post #105

Are there any case studies where microservices went well? From an end user perspective, Netflix runs in “constantly degraded” mod. From an engineering perspective, they track “number of successful stream starts”, instead of percentage of the time 100% of their services are working. That’s a huge red flag. As a researcher, the monitoring and fault-propagation / modeling work they’ve done to get it to stay up at all is…

By sheer number of attempts somebody probably got good results with microservices somewhere.

Netflix runs quite well in practice. I think they do redundant service calls, what is the only minimally sane way to develop a distributed system. The funny thing is that I have never seen any serious discussion of redundant calls, except for it being implicit on practical designs on the anecdotal "how it works for us" articles that pop once in a while. Most times people won't even discuss redundant servers. I imagine everybody thinks it's obvious, and well, I would agree, except for the fact that most people I see do not think so.

But well, Netflix couldn't avoid having a distributed system, so they aren't really representative for nearly anybody.

Re: Why Segment Went Back to a Monolith

#127

Post by herval » Wed, Apr 29, 2020, 2:45 PM UTC

cturner wrote: post #89

yowlingcat wrote: post #79

Earlier quoted context omitted.

What is your alternative? Tying "the existence of internal software components to people's livelihoods" across the expanse of the entire codebase is the only remotely effective approach I've seen to scaling the SDLC at scale.

"What is your alternative?" Aggressively small teams, with no hands-off middle-management layer. You can build massive capability around a small number of well-managed message-backbones and a single codebase. By keeping the number of hands small and the structure flat, you force high standards. (Skilled staff won't tolerate distractions caused by bad engineering or inadequate automation.) Heuristic for analysing firm…

Is there any example where this works (articles, presentations, etc)? In particular, anywhere with more than a couple dozen developers?

Re: Why Segment Went Back to a Monolith

#128

Post by marcosdumay » Wed, Apr 29, 2020, 2:46 PM UTC

Autowired wrote: post #90

pjc50 wrote: post #66

Earlier quoted context omitted.

This is really a time-of-binding argument; the difference between a "library" and a "service" is that one is in-process and accessed over function calls, and the other is out-process and accessed over RPC. If you change code that other services are using, you can break those other services. No way round that.

While that is true, a microservices architecture can (and in my opinion should) rely on messaging and account for message schema evolution. Dependencies between services should be way less coupling than dependencies between an application and a library.

A library API can rely on versioning and account for schema evolution too. Even different versions can coexist if you decide that's important from the beginning (what is the same requirement as with services).

The only real difference is that services have a slow serialized network interface that fails 4 or 5 orders of magnitude more often than libraries, but can migrate over memory domains.

Re: Why Segment Went Back to a Monolith

#129

Post by mcansky » Wed, Apr 29, 2020, 2:47 PM UTC

there is a related post on Segment's blog : https://segment.com/blog/goodbye-microservices/ which helps to get a bit more details and contexts

update : and related video https://www.youtube.com/watch?v=lv5o3qnQu5w

Re: Why Segment Went Back to a Monolith

#130

Post by fennecfoxen » Wed, Apr 29, 2020, 2:47 PM UTC

BareNakedCoder wrote: post #103

axegon_ wrote: post #95

Earlier quoted context omitted.

Mmmmyes and no. Depending on the size of your project, that may not be the case. I've had to work with two titans of monoliths, maintained by relatively small teams(anywhere between 2 and 6-7 people for several million lines of code). At some point managing a codebase this big within a single project becomes a huge burden, for both developers and even more so for those who develop and do code-reviews(first hand exper…

Good monoliths are highly modularized. But it's a whole different thing to package up a module as a separately deployable unit for external "public" use (external to your app, that is, not your company). I'm just curious to know, when you said "the easiest and most sensible thing to do is chunk out large parts of the project and put them aside as a microservice" ... were these chunks separately deployable units for e…

I think this is actually one of the reasons that microservices became a thing to begin with: teams wouldn't actually apply engineering best practices.

Microservices actually make you encapsulate your code, at least within the microservices, because you can't call out to it directly. They don't necessarily force you to implement the single responsibility principle, but they do a good job of pushing you. Microservices implement a service-locator pattern through DNS or web routing, one form of the dependency inversion principle. Microservices make you pass data around as entities, instead of Active Record instances.

The price for this sort of thing is very steep, though; distributed systems are inherently icky, harder to trace, and more prone to failure, and besides this, you've added network overhead to each service call.

I wish more engineering teams would consider spending half the effort of microservices on simply disciplining their monoliths. They might get somewhere...