Keep the monolith, but split the workloads • Hacker News Board

Re: Keep the monolith, but split the workloads

#71

Post by zimpenfish » Mon, Apr 24, 2023, 12:22 PM UTC

lawrjone wrote: post #40

KronisLV wrote: post #36

I quite like the article and the advice it presents, building what I'd call modular monoliths (that can have modules be enabled or disabled based on feature flags) is indeed a good approach for both increasing resiliency and decreasing the blast radius of various issues. However, this bit stuck out to me: > When a bad Pub/Sub message was pulled into the binary, an unhandled panic would crash the entire app, meaning w…

Have you worked with Go codebases before? Standard practice is to wrap all your entrypoints - the start of a web request, the moment you begin to run a job - in a defer recover() which will catch panics. Sadly, recover won’t apply to any subsequently goroutine’d work. That means even if your entrypoints recover, if anything go func()s down the call stack, then if that function then panics it will bring down the entir…

> Have you worked with Go codebases before?

Several but ...

> Standard practice is to wrap all your entrypoints [...] in a defer recover()

... I've never seen that. Is there some literature pointing to this as best practice?

Re: Keep the monolith, but split the workloads

#72

Post by Jupe » Mon, Apr 24, 2023, 12:26 PM UTC

lawrjone wrote: post #2

Author here, thanks for sharing! This is a strategy I’ve used across many projects in several languages, from big Rails apps to the Go monolith we deploy at incident.io today. It’s just one way you can make a monolith much more robust without moving into separate services, which helps you keep the benefits of a monolithic application for longer. Hope people find it interesting.

While I completely agree that the efforts to build and maintain a set of micro-services are often better leveraged by a single monolith (even one that has several "run modes" as you've done here), a few questions inevitably come up:

How do you coordinate the efforts of 200 engineers on a single repo/code base?

Do engineers frequently get into long/drawn-out merge sessions as common code may be modified by a number of engineers who are all trying to merge around the same time? This is actually one of the reasons I really like GOLANG: "A little copying is better than a little dependency."

Re: Keep the monolith, but split the workloads

#73

Post by ibejoeb » Mon, Apr 24, 2023, 12:26 PM UTC

slig wrote: post #25

KyeRussell wrote: post #7

Earlier quoted context omitted.

Ditto in Django + Celery

Anyone using a more modern/lightweight alternative to Celery?

If you're using PostgreSQL, then

django-postgres-queue: https://github.com/gavinwahl/django-postgres-queue

procrastinate: https://github.com/procrastinate-org/procrastinate/

Re: Keep the monolith, but split the workloads

#74

Post by andrewflnr » Mon, Apr 24, 2023, 12:33 PM UTC

speedbird wrote: post #67

I just don’t get the obsession with micro services (by with I mean a complete tech stack for each service). I can only see the relevance in a sub optimal org structure - huge org, merger etc. SOA, sure (hello CICS :-)). DB sharding if needed for load / resources. Otherwise logical division into sets of related services that can be their own little monolith or go full monorepo. Perhaps I'm missing something …

The difference with SOA being that all the services use roughly the same stack?

Re: Keep the monolith, but split the workloads

#75

Post by intelVISA » Mon, Apr 24, 2023, 12:41 PM UTC

devoutsalsa wrote: post #27

dumpster_fire wrote: post #11

One critique I have is that this presents a binary option of either full monolith and microservices. The truth is somewhere in between where you have dependent services large enough to warrant being their own monolith being split off. Breaking up of a monolith is almost never (anecdotal observation after 15 years of seeing this argument surface in every company I've been in) a technical need, but a combination of org…

A monolith doesn’t have to be one giant ball of mud. It can be discrete, well-factored services all by itself. I recently worked on decomposing a monolith into micro services, but it felt like we were just spreading one big problem over multiple services. All of the services ended up being tightly coupled, even the the goal was to avoid that. We created a macrolith.

Agreed, transforming a monolith into a 'distributed monolith' just creates a different set of even worse problems, the root issues are still unsolved.

Still, thanks to AWS it's vogue, expensive and probably keeps half of us employed when we have to untangle it all!

Re: Keep the monolith, but split the workloads

#76

Post by taneq » Mon, Apr 24, 2023, 12:41 PM UTC

devoutsalsa wrote: post #27

dumpster_fire wrote: post #11

One critique I have is that this presents a binary option of either full monolith and microservices. The truth is somewhere in between where you have dependent services large enough to warrant being their own monolith being split off. Breaking up of a monolith is almost never (anecdotal observation after 15 years of seeing this argument surface in every company I've been in) a technical need, but a combination of org…

A monolith doesn’t have to be one giant ball of mud. It can be discrete, well-factored services all by itself. I recently worked on decomposing a monolith into micro services, but it felt like we were just spreading one big problem over multiple services. All of the services ended up being tightly coupled, even the the goal was to avoid that. We created a macrolith.

Exactly! All this debate seems to stem from people being forced to work with a big ball of mud and then concluding that no one process should ever be allowed to become that big, because big processes are balls of mud. And so, having missed the real lesson (which isn’t ‘don’t make it big’ but rather ‘don’t make it out of mud’!) they build a big ball of little balls of mud.

Re: Keep the monolith, but split the workloads

#77

Post by ohgodplsno » Mon, Apr 24, 2023, 12:42 PM UTC

hardware2win wrote: post #66

iamflimflam1 wrote: post #43

Earlier quoted context omitted.

Are you doing “extreme” micro services? A esperare database for each micro service, one endpoint for each microservice etc.. or the slightly more sensible - let’s split the monolith up into obviously independent system (very much like the article talks about).

>extreme microservices >separate databases Isnt it like the basic thing?

If your database isn't the performance issue and those two services spent 99% of the response time on calculating the 15755th digit of pi, not separating the database isn't a problem. Similarly, if both of your services need to fetch some user information, but you haven't created a user information service with its own database they could both talk to, having a shared database is fine.

Splitting each and every service with its own database, talking to other services to access their database by default is cargo-cult behavior. All it's going to do is make it hell for you to work on those services.

Re: Keep the monolith, but split the workloads

#78

Post by slig » Mon, Apr 24, 2023, 12:44 PM UTC

ibejoeb wrote: post #73

slig wrote: post #25

Earlier quoted context omitted.

Anyone using a more modern/lightweight alternative to Celery?

If you're using PostgreSQL, then django-postgres-queue: https://github.com/gavinwahl/django-postgres-queue procrastinate: https://github.com/procrastinate-org/procrastinate/

Fantastic, I'm. Thanks.

Re: Keep the monolith, but split the workloads

#79

Post by ianpurton » Mon, Apr 24, 2023, 12:48 PM UTC

Monolith is not a best practice.

With most web applications you'll end up with a web part and at some point some crob jobs. In this case they also have a service that processes a queue.

They should in be in the same repo, that's a Monorepo.

A web app with some cronjobs is best deployed as separate executables (ideally docker containers). In this case they had everything in one exe, including threads to run the queue etc.

So it looks like there solution was more complicated by aiming to be a Monolith.

Re: Keep the monolith, but split the workloads

#80

Post by lawrjone » Mon, Apr 24, 2023, 12:53 PM UTC

zimpenfish wrote: post #71

lawrjone wrote: post #40

Earlier quoted context omitted.

Have you worked with Go codebases before? Standard practice is to wrap all your entrypoints - the start of a web request, the moment you begin to run a job - in a defer recover() which will catch panics. Sadly, recover won’t apply to any subsequently goroutine’d work. That means even if your entrypoints recover, if anything go func()s down the call stack, then if that function then panics it will bring down the entir…

> Have you worked with Go codebases before? Several but ... > Standard practice is to wrap all your entrypoints [...] in a defer recover() ... I've never seen that. Is there some literature pointing to this as best practice?

You’ll find some libraries do this for you, such as HTTP servers.

They do this because if your server code makes a mistake such as accessing a nil pointer, a segfault or panic would bring down the entire process. That’s why you want to recover(), to avoid a your process dying.