Why Segment Went Back to a Monolith • Hacker News Board

Re: Why Segment Went Back to a Monolith

#101

Post by gridlockd » Wed, Apr 29, 2020, 1:37 PM UTC

alkonaut wrote: post #64

klohto wrote: post #60

Earlier quoted context omitted.

Because if you share something (like auth for example), you should have microservice for that. The question is not about duplicate code, but about duplicate libraries that handle the same thing. Decoupling the auth process into separate microservice removes the bottleneck.

Eventually you'll have a service to format phone numbers in the format that the company needs to be standard across all services. If you don't want to do that, then you need a simple shared library for that. The problem is that there is no easy way to draw the line between "this is obviously a trivial library function we should just link into our code" and "this is something we can't share because it would create fri…

> Auth is obviously a "service" but phone number formatting as a service seems extreme.

You clearly haven't finished drinking your Kool-Aid yet.

Re: Why Segment Went Back to a Monolith

#102

Post by jmilloy » Wed, Apr 29, 2020, 1:40 PM UTC

kkapelon wrote: post #46

>Shared libraries were created to provide behavior that was similar for all workers. However, this created a new bottleneck, where changes to the shared code could require a week of developer effort, mostly due to testing constraints. That is a big red flag. Microservices that suffer from shared code changes are not really microservices, but a distributed monolith instead.

That just sounds like the shared libraries needed to make breaking changes less often. If you're going to make changes to core code, it's going to take time to get everything up to date no matter how your code is organized. In other words, shared code needs to be treated just like a third-party library/service (both from the developers and users points of view).

Re: Why Segment Went Back to a Monolith

#103

Post by BareNakedCoder » Wed, Apr 29, 2020, 1:45 PM UTC

axegon_ wrote: post #95

nickcw wrote: post #19

I think that the problem here was that they were fighting against Conway's Law: https://en.wikipedia.org/wiki/Conway%27s_law > Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure. I think microservices work well in organizations that are big enough to have a team per microservice. However if you've just split your monol…

Mmmmyes and no. Depending on the size of your project, that may not be the case. I've had to work with two titans of monoliths, maintained by relatively small teams(anywhere between 2 and 6-7 people for several million lines of code). At some point managing a codebase this big within a single project becomes a huge burden, for both developers and even more so for those who develop and do code-reviews(first hand exper…

Good monoliths are highly modularized. But it's a whole different thing to package up a module as a separately deployable unit for external "public" use (external to your app, that is, not your company).

I'm just curious to know, when you said "the easiest and most sensible thing to do is chunk out large parts of the project and put them aside as a microservice" ... were these chunks separately deployable units for external "public" use.

Re: Why Segment Went Back to a Monolith

#104

Post by mark_l_watson » Wed, Apr 29, 2020, 1:45 PM UTC

I have always viewed using many micro services as something that adds complexity, something to be used when necessary.

I started working remotely as a consultant in the early 2000s when my wife and I moved to a remote area. I had several development jobs that used the same monolith pattern: I would embed everything in a web app using Apache Tomcat, taking advantage of work threads for background tasks. The only external services were a database and crontab settings to frequently snapshot databases. This pattern was so easy to code to, so easy to debug and deal with any runtime problems. One customer reported that a system ran without stopping for six years (ouch, no OS upgrades??) until they restarted it on a larger server.

Micro services can be great, but not always the best choice when horizontal scaling is not required.

Re: Why Segment Went Back to a Monolith

#105

Post by hedora » Wed, Apr 29, 2020, 1:47 PM UTC

Are there any case studies where microservices went well?

From an end user perspective, Netflix runs in “constantly degraded” mod.

From an engineering perspective, they track “number of successful stream starts”, instead of percentage of the time 100% of their services are working. That’s a huge red flag.

As a researcher, the monitoring and fault-propagation / modeling work they’ve done to get it to stay up at all is impressive, but it’s not clear all of that tooling would be necessary if they didn’t have to reason about N^2 fault tolerance scenarios, where N = 100’s of microservices. That’s on the order of one fault tolerance scenario for each atom in the universe.

Re: Why Segment Went Back to a Monolith

#106

Post by gen220 » Wed, Apr 29, 2020, 2:00 PM UTC

thedevopsguy wrote: post #53

Without knowing more about their architecture it is difficult to comment beyond the conclusion Alexandra Noonan came to, stated at the beginning of the article. It looks like to me that the architectural assumptions were changing too quickly due to the demands of a fast growing business. Having all their code in a single repository means that they can control dependencies, versioning and deployment centrally, it give…

I totally agree with you.

I think this article is more evidence against the credibility of multi-repo than against "microservices".

Anecdotally, my current place of work has grown to about 200 engineers, maintains a monorepo, and hundreds of deployed cron jobs, ad-hoc jobs, and "microservices". We have none of the problems discussed here. We invest maybe 20 eng weeks a year in monorepo-specific tooling, and perhaps another 30 eng weeks per year in "microservices"-tooling.

Re: Why Segment Went Back to a Monolith

#107

Post by neya » Wed, Apr 29, 2020, 2:00 PM UTC

Microservice is not an organizational problem. Microservices is a design problem. If you implement DDD (Domain Driven Design) first to your application and then start to design the application around your DDD concept, then, it might work.

But, it's extremely hard even to do that. Microservices simply complicates things if any of your domains need to share code with each other. Many DDD paradigms exist to address this, but none are practical. For example, authentication related code. IF one domain sets a cookie and the other one has to rely on that to keep the user (a shared model between the two domains) authenticated, then this means, you need to duplicate code bases across two domains or in the very least put them into some sort of shared helper/library, which DDD is kind of against.

That's why it totally makes sense to go Monolith first and really identify the parts of your application that are slowing you down either development wise, testing wise or performance wise and put them into separate contexts.

Phoenix actually does Microservices right. From all the way to scaffold generation to instructing best practices on keeping your domains properly separated. But even then, I've burnt my finger many a times trying to write simple CMS solutions into mutliple microservices then going back to monoliths again.

Re: Why Segment Went Back to a Monolith

#108

Post by caust1c » Wed, Apr 29, 2020, 2:07 PM UTC

Autowired wrote: post #90

pjc50 wrote: post #66

Earlier quoted context omitted.

This is really a time-of-binding argument; the difference between a "library" and a "service" is that one is in-process and accessed over function calls, and the other is out-process and accessed over RPC. If you change code that other services are using, you can break those other services. No way round that.

While that is true, a microservices architecture can (and in my opinion should) rely on messaging and account for message schema evolution. Dependencies between services should be way less coupling than dependencies between an application and a library.

Schema evolution is just as big of a dependency hell as managing direct library dependencies. With a monolithic architecture, a lot of those concerns are contained within the context of a single repo, and can be tested much more easily than with many repos.

Re: Why Segment Went Back to a Monolith

#109

Post by 87zuhjkas » Wed, Apr 29, 2020, 2:10 PM UTC

hedora wrote: post #105

Are there any case studies where microservices went well? From an end user perspective, Netflix runs in “constantly degraded” mod. From an engineering perspective, they track “number of successful stream starts”, instead of percentage of the time 100% of their services are working. That’s a huge red flag. As a researcher, the monitoring and fault-propagation / modeling work they’ve done to get it to stay up at all is…

Well, lets assume N = 999, then N^2 = 998001, so nowhere near the number of atoms in the universe, which is estimated to be about 10^80.

Re: Why Segment Went Back to a Monolith

#110

Post by organsnyder » Wed, Apr 29, 2020, 2:11 PM UTC

kitd wrote: post #31

nickcw wrote: post #19

I think that the problem here was that they were fighting against Conway's Law: https://en.wikipedia.org/wiki/Conway%27s_law > Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure. I think microservices work well in organizations that are big enough to have a team per microservice. However if you've just split your monol…

As with a lot of things, it comes down to communication. Between teams, and between the services they write. Which is just another expression of Conway's Law. IIRC Fred Brooks pointed out that the # of bugs in a system correlates closely with the # of lines of communication within and between the teams. Joshua Bloch recommends in "Effective Java" that, if possible, 3 potential clients should participate in the design…

> So a well-designed interface or OpenAPI spec is worth its weight in gold.

When I worked on a SOA team, I tried to begin any new effort (whether a new API or modification to existing API) solely discussing the API contract. It was (ideally) high-level enough that business analysts and project managers would understand it, and it helped to guide us away from getting mired in implementation discussions too early.

At that organization, we rarely had the opportunity to involve multiple customers at the same time during design discussions (we were typically engaged to help a specific consumer implement a specific feature), but the institutional memory in the SOA team helped us to keep in mind existing/potential other users of each particular webservice.