Keep the monolith, but split the workloads • Hacker News Board

Re: Keep the monolith, but split the workloads

#91

Post by revskill » Mon, Apr 24, 2023, 1:31 PM UTC

brentjanderson wrote: post #68

cle wrote: post #64

Earlier quoted context omitted.

To me, it's unclear what the best solution is here. Other languages solve this differently with tradeoffs, e.g. Java defaults to threads silently dying when an exception isn't caught. Your program will continue to run, but it's probably in some undefined state at that point. There are mechanisms for propagating exceptions elsewhere, but they have to be explicitly set up (like in Go). You can set a default uncaught ex…

Erlang/Elixir have a great story here: “let it crash”. Because each slice of activity in an application is wrapped in its own process (think single threaded loop but you can run a million at a time, almost free to create and destroy), if it crashes it only takes down that web request/process. Recovery mechanisms are built in to get back to a know good state.

Pm2 for nodejs is the same.

Re: Keep the monolith, but split the workloads

#92

Post by jameshart » Mon, Apr 24, 2023, 1:34 PM UTC

* edit: I am an idiot who can’t recognize golang code (facepalm)

So it’s a Ruby* app, which means code you put in the container but never use doesn’t do you any harm - it doesn’t bloat a binary or take up space in memory.

but…

… whenever you make a change in one of your pub-sub handlers, you still have to redeploy your web workers.

… code and dependencies for your pub-sub handlers is sitting idle in your web workers, increasing your security risk and attack surface

Why not just turn this into a monorepo that publishes three artifacts? Having just one container image seems like a weird thing to optimize for

Re: Keep the monolith, but split the workloads

#93

Post by jameshart » Mon, Apr 24, 2023, 1:36 PM UTC

andrewflnr wrote: post #74

speedbird wrote: post #67

I just don’t get the obsession with micro services (by with I mean a complete tech stack for each service). I can only see the relevance in a sub optimal org structure - huge org, merger etc. SOA, sure (hello CICS :-)). DB sharding if needed for load / resources. Otherwise logical division into sets of related services that can be their own little monolith or go full monorepo. Perhaps I'm missing something …

The difference with SOA being that all the services use roughly the same stack?

I always think of the difference between SOA and microservices as being that in SOA the services often share access to the same database.

Re: Keep the monolith, but split the workloads

#94

Post by 0xbadcafebee » Mon, Apr 24, 2023, 1:37 PM UTC

I never see a blog post looking at the specific requirements of an application and fitting the architecture, data model, and implementation to those requirements. "Monolith vs Microservices" is like saying "18-wheeler vs 20 Toyota Corollas". There are other forms of transportation, and using just one mode may not serve your business well.

By the way, it's 2023... if you have a greenfield project, you should be using event-driven data processing. We got rid of the horse and buggy, let's please get rid of cron jobs.

Re: Keep the monolith, but split the workloads

#95

Post by mjr00 » Mon, Apr 24, 2023, 1:42 PM UTC

devoutsalsa wrote: post #27

dumpster_fire wrote: post #11

One critique I have is that this presents a binary option of either full monolith and microservices. The truth is somewhere in between where you have dependent services large enough to warrant being their own monolith being split off. Breaking up of a monolith is almost never (anecdotal observation after 15 years of seeing this argument surface in every company I've been in) a technical need, but a combination of org…

A monolith doesn’t have to be one giant ball of mud. It can be discrete, well-factored services all by itself. I recently worked on decomposing a monolith into micro services, but it felt like we were just spreading one big problem over multiple services. All of the services ended up being tightly coupled, even the the goal was to avoid that. We created a macrolith.

It's not about monoliths always being a ball of mud. Even the most well-composed monolith still has problems with teams wanting to do conflicting release cycles, needing clearer ownership over who has responsibility for what part of the codebase, and knowing who should be responsible for on-call for which services. And there is, of course, dependency hell, since everything in your monolith probably should depend on the same version of third-party libraries.

Re: Keep the monolith, but split the workloads

#96

Post by krab » Mon, Apr 24, 2023, 1:45 PM UTC

jameshart wrote: post #92

* edit: I am an idiot who can’t recognize golang code (facepalm) So it’s a Ruby* app, which means code you put in the container but never use doesn’t do you any harm - it doesn’t bloat a binary or take up space in memory. but… … whenever you make a change in one of your pub-sub handlers, you still have to redeploy your web workers. … code and dependencies for your pub-sub handlers is sitting idle in your web workers,…

> … whenever you make a change in one of your pub-sub handlers, you still have to redeploy your web workers.

We have a similar model with Django. We deploy at least four times a day, so it's much easier to just redeploy everything than reason about what needs and what doesn't need to be redeployed.

Re: Keep the monolith, but split the workloads

#97

Post by ohgodplsno » Mon, Apr 24, 2023, 1:48 PM UTC

hardware2win wrote: post #81

ohgodplsno wrote: post #77

Earlier quoted context omitted.

If your database isn't the performance issue and those two services spent 99% of the response time on calculating the 15755th digit of pi, not separating the database isn't a problem. Similarly, if both of your services need to fetch some user information, but you haven't created a user information service with its own database they could both talk to, having a shared database is fine. Splitting each and every servic…

But if 1 database dies, then all microservices dont work So why are you even using them? The benefit of microservices is also reliability, which you just threw away Isnt this basically distributed monolith? So you combined bad things of both worlds! Single point of failure of monolith And deployment difficulties and need for saga like patterns to deal with network trickiness from distributed designs Whats the point?

Once again, cargo-cultish reasoning, where it's all or nothing.

1/ Replication & read-only copies for resilience exist, to allow you to function in degraded state if your database goes down.

2/ If the database for one service goes down, any service that calls it ends up being down anyways. No, being able to respond with a 503 that says that megatron-service is down and you can't fetch the data isn't different from responding with a regular 500 saying that the database is down and you can't fetch the data..

3/ All of your calls do not necessarily end up calling the database. Your service starts even if the DB is down, some calls will simply fail. You're providing a degraded service.

4/ If they need similar data, you really don't want to have two databases holding that data. Because I can promise you, the cost of having to handle replication, de-duplication, synchronization, configuration, GDPR compliance, etc on X databases is infinitely higher than having a database where you simply turn on a write replica that fails over.

5/ SPOF-fear is bullshit, you always have a single point of failure in your stuff. 2FA service is down ? Sure, your entire service isn't dead, but hey, users can't log in, sounds like a pretty fucking big point of failure. Microservices just distributes your single point of failure into a dozen, equally bad, equally hard to track down problems. And even in a monolith, you can easily work around these problems: what kind of fucking code have you seen that a whole server will not start because it can't connect to a database ?

6/ You do not need your servers to be distributed. That is bullshit, and if you are at the scale where you actually need to, it'll be the least of your problems.

Re: Keep the monolith, but split the workloads

#98

Post by twosdai » Mon, Apr 24, 2023, 1:48 PM UTC

jameshart wrote: post #92

* edit: I am an idiot who can’t recognize golang code (facepalm) So it’s a Ruby* app, which means code you put in the container but never use doesn’t do you any harm - it doesn’t bloat a binary or take up space in memory. but… … whenever you make a change in one of your pub-sub handlers, you still have to redeploy your web workers. … code and dependencies for your pub-sub handlers is sitting idle in your web workers,…

I think that publishing 3 artifacts causes a number of other changes which expand the difficulty of management of the backend of the app. While they do allow, hopefully, independent scaling and seperate recovery processes. Separate artifacts also make it a bit more difficult to do local development, and deployments become more complex so it's not a completely clean tradeoff.

I know systems kubernetes / dockerswarm which handle the management of multiple artifacts but that is now another tool needed in order to do development.

So I guess what I'm saying is that "just" publishing multiple artifacts has more knock-on effects which need to be considered. Eventhough I do agree with you that it's probably for the best in the long term if these processes are in seperate containers.

Re: Keep the monolith, but split the workloads

#99

Post by dboreham » Mon, Apr 24, 2023, 1:51 PM UTC

cle wrote: post #64

lawrjone wrote: post #40

Earlier quoted context omitted.

Have you worked with Go codebases before? Standard practice is to wrap all your entrypoints - the start of a web request, the moment you begin to run a job - in a defer recover() which will catch panics. Sadly, recover won’t apply to any subsequently goroutine’d work. That means even if your entrypoints recover, if anything go func()s down the call stack, then if that function then panics it will bring down the entir…

To me, it's unclear what the best solution is here. Other languages solve this differently with tradeoffs, e.g. Java defaults to threads silently dying when an exception isn't caught. Your program will continue to run, but it's probably in some undefined state at that point. There are mechanisms for propagating exceptions elsewhere, but they have to be explicitly set up (like in Go). You can set a default uncaught ex…

> when an exception isn't caught

Not catching all exceptions is a glaring P0 bug.

Re: Keep the monolith, but split the workloads

#100

Post by krab » Mon, Apr 24, 2023, 1:54 PM UTC

Jupe wrote: post #72

lawrjone wrote: post #2

Author here, thanks for sharing! This is a strategy I’ve used across many projects in several languages, from big Rails apps to the Go monolith we deploy at incident.io today. It’s just one way you can make a monolith much more robust without moving into separate services, which helps you keep the benefits of a monolithic application for longer. Hope people find it interesting.

While I completely agree that the efforts to build and maintain a set of micro-services are often better leveraged by a single monolith (even one that has several "run modes" as you've done here), a few questions inevitably come up: How do you coordinate the efforts of 200 engineers on a single repo/code base? Do engineers frequently get into long/drawn-out merge sessions as common code may be modified by a number of…

I have worked in a company with a monolith and about ten teams working on it. This is what helped:

- Merging was automated (a robot tried to run tests against fresh master and merge only if green).

- Deploy was fully automated and limited to working hours.

- We added tests for problematic parts. For example static analysis for database migrations to prevent only safe actions in an automated fashion.

However, if something goes wrong in some component, you have to revert and stop the deploys for everyone which sucks. I'd say around 8 - 10 deploys per day, it makes sense to start splitting the components or at least not adding new teams to the same monolith.