Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Keep the monolith, but split the workloads

incident.io

Post reply on HN
Print view View original

121–130 of 161 posts

Re: Keep the monolith, but split the workloads

#121

Post by [deleted] »

jrockway wrote: post #119
lawrjone wrote: post #80

Earlier quoted context omitted.

You’ll find some libraries do this for you, such as HTTP servers. They do this because if your server code makes a mistake such as accessing a nil pointer, a segfault or panic would bring down the entire process. That’s why you want to recover(), to avoid a your process dying.

I mean, net/http does it. That's the standard library. A go convention that I've made up, or that is perhaps a real one, is to always look at the standard library for guidance on how to write Go. net/http suffers a little bit from being a very early library and thus you might not want to emulate its API surface. But in general, the Go team thought "you know what every HTTP handler in Go needs? recovery from panics" a…

[deleted]

Re: Keep the monolith, but split the workloads

#122

Post by dontlaugh »

gregwebs wrote: post #37

It’s odd that there’s still one binary. You can have a mostly monolithic library but produce a web binary and a worker binary, etc. Go can produce them efficiently with a single build command.

Agreed, you can even put all the binaries in the same image.

Ultimately not a big deal.

Re: Keep the monolith, but split the workloads

#123

Post by mikeryan »

Olphs wrote: post #34
dumpster_fire wrote: post #11

One critique I have is that this presents a binary option of either full monolith and microservices. The truth is somewhere in between where you have dependent services large enough to warrant being their own monolith being split off. Breaking up of a monolith is almost never (anecdotal observation after 15 years of seeing this argument surface in every company I've been in) a technical need, but a combination of org…

Do you think it's good idea to migrate from monolith to microservices before scaling the team? We are essentially doing this right now, I personally don't think the teams are yet large enough to warrant this migration, but leadership says that we can't scale further before we have microservices. Imo most of the problems could be solved with increased testing and knowledge about the system. The tech also could definit…

can't scale further before we have microservices.

Scale what? Load (Seems not)

Scale the team? This is often an overlooked aspect of microservices you're able to isolate the domain knowledge to smaller services and hire more specialized engineers who can focus on a specific silo of the app and only have to work to discrete APIs.

Scale features? There's a legit argument to be made that if you see yourself going to a microservice architecture in the future based on your current codebase or product roadmap that doing it sooner then later is always going to be preferable. The bigger the monolith the harder it is to unwind. It could also be that there's some tech debt that's something of a limiting factor, making the change in architecture could be an opportunity to address some of that.

Re: Keep the monolith, but split the workloads

#124

Post by jameshart »

lawrjone wrote: post #115
jameshart wrote: post #105

Earlier quoted context omitted.

Apologies - read the ‘Ruby monolith’ up top and then glossed over the code without noticing the language mismatch. Mondays, am I right? I don’t understand how bundling three sets of functionality into one binary reduces build time or artifact size. Surely the binary containing all the web and pubsub and scheduled tasks is strictly bigger than the binary for just the web would be, and takes longer to build and test th…

No problem, easy to skim by it! In terms of how this reduces artifact size: Go statically compiles everything, and the majority of that weight is from dependencies. Compiling into three different binaries would: 1. Increase build time, because even while the dependencies are cached from each subsequent build, linking isn't free. Takes about 20s to link each binary, so it adds about 40s+ to the build to create them tw…

You only have to rebuild the artifacts that are affected by each change.

And each change (assuming CI/CD) only causes the changes build artifacts to be pushed to prod.

So the amortized build/push cost depends on how often you are making changes that affect multiple artifacts versus fewer artifacts.

If an individual change only affects either the web binary or one of the cron job binaries then those build times will be shorter than they would be to build the big binary, and the amount of data pushed to prod will be smaller.

If an individual change affects multiple binaries then building all of them will take longer, and more data will need to be pushed (although - maybe it nets out the same in terms of fa out to multiple servers - your cron servers only need the cron binary, your web servers only need the web binary)

So, if the vast majority of the time you are only impacting a single component build, you save build time and deploy bytes by building smaller artifacts.

At the cost that making cross-system changes requires a slightly longer build time.

Re: Keep the monolith, but split the workloads

#125

Post by pphysch »

ahofmann wrote: post #15

One often finds derogatory remarks about PHP, or that the popularity of the language is declining. Interestingly, the concept of PHP prevents exactly such problems, as a monolith written in PHP only ever executes the code paths that are necessary for the respective workload. The failure that the Rails app experienced in the post simply wouldn't have happened with PHP. Especially for web applications, PHP's concept of…

> Interestingly, the concept of PHP prevents exactly such problems, as a monolith written in PHP only ever executes the code paths that are necessary for the respective workload.

This is just lovely when you have hundreds of PHP endpoints written by your predecessors and each endpoint has rewritten an arbitrary slice of the stack (usually data model layer) because there is no common code path required. Refactoring anything below the html layer becomes impossible.

In fact, calling it a software monolith is misleading, because each PHP script is its own little microservice with poorly-defined API boundaries.

Re: Keep the monolith, but split the workloads

#126

Post by jameshart »

mjr00 wrote: post #95
devoutsalsa wrote: post #27

Earlier quoted context omitted.

A monolith doesn’t have to be one giant ball of mud. It can be discrete, well-factored services all by itself. I recently worked on decomposing a monolith into micro services, but it felt like we were just spreading one big problem over multiple services. All of the services ended up being tightly coupled, even the the goal was to avoid that. We created a macrolith.

It's not about monoliths always being a ball of mud. Even the most well-composed monolith still has problems with teams wanting to do conflicting release cycles, needing clearer ownership over who has responsibility for what part of the codebase, and knowing who should be responsible for on-call for which services. And there is, of course, dependency hell, since everything in your monolith probably should depend on t…

> everything in your monolith probably should depend on the same version of third-party libraries.

And definitely runs on the same underlying language/compiler/runtime version

Re: Keep the monolith, but split the workloads

#127

Post by rektide »

Front end gateways make this so easy. Stand up the service a couple of times, then send different routes to different pools. This is the easiest implementation of the Bulkhead Pattern one can see, it's great. https://learn.microsoft.com/en-us/azure/architecture/pattern...

I've had so many jobs where we either have some awful slow routes, or we have some latency critical routes. Thusfar I have yet to convince a single company what an incredible & vital win this would be, to create different service pools for certain classes of routes, but wow have I tried to make it happen & man what a win it would be for users.

Re: Keep the monolith, but split the workloads

#128

Post by Lightbody »

We do something similar with Reclaim.ai:

It's one monolith Java repo, split into two major workloads that have the same runtime code but are configured differently. In our case, the two workflows are "api" and "jobs". This allows for operational issues on, say, jobs while still allowing user-facing APIs to be less likely to be impacted. The source code itself has many modules, but we deploy it as a single unified runtime.

We're very happy with this approach and so far have only been tempted a few times to break out micro/mini-services. I suspect we will eventually break out a few services, but by the time we do, it'll be for very good reasons.

Re: Keep the monolith, but split the workloads

#129

Post by heywhatupboys »

kubanczyk wrote: post #118
lawrjone wrote: post #80

Earlier quoted context omitted.

You’ll find some libraries do this for you, such as HTTP servers. They do this because if your server code makes a mistake such as accessing a nil pointer, a segfault or panic would bring down the entire process. That’s why you want to recover(), to avoid a your process dying.

> They do this because if your server code makes a mistake This is neither good or best practice. My take: - Know that there are simple-mistake panics, and internal-state-just-went-bonkers panics. For the latter, you can guess the boundary of impact (one request, one connection, one job, one userID, one process, ...) but exit(1) is much more reliable than guessing. - Tests can easily catch simple mistakes like access…

> - Tests can easily catch simple mistakes like accessing a nil pointer.

no they can't . this is exactly what is hard to test for, complex state that can occur by some combination of many variables on many values

Re: Keep the monolith, but split the workloads

#130

Post by marcosdumay »

KronisLV wrote: post #36

I quite like the article and the advice it presents, building what I'd call modular monoliths (that can have modules be enabled or disabled based on feature flags) is indeed a good approach for both increasing resiliency and decreasing the blast radius of various issues. However, this bit stuck out to me: > When a bad Pub/Sub message was pulled into the binary, an unhandled panic would crash the entire app, meaning w…

> There's always some error handling logic wrapping around the request handling

Yeah, some languages make this easier than others.

The way you described it made me think it was Rust, a language where those handlers are not trivial, but not incredibly hard either. But it seems that Go developers have it worse (no big surprise here).

Of course, the champions on unhandled failures are always C and C++, where almost any issue is impossible to recover from. It's no coincidence that those are low level languages (and Go).

When you move from the more controlled languages into those, you tend to also move from in-language error recovery to system-wide error recovery.

Post reply on HN