Isn’t a monolith with split workloads just a monorepo?
Keep the monolith, but split the workloads
51–60 of 161 posts
Re: Keep the monolith, but split the workloads
#52One critique I have is that this presents a binary option of either full monolith and microservices. The truth is somewhere in between where you have dependent services large enough to warrant being their own monolith being split off. Breaking up of a monolith is almost never (anecdotal observation after 15 years of seeing this argument surface in every company I've been in) a technical need, but a combination of org…
And one technical / business side that's very valid: error-proofing the service and database. It is to prevent the database to be populated incorrectly, such as financial data, that you proof it behind a set of well-defined API, to prevent any leakage of sensitive data (PII or credit-card info) and to be easily audited.
Re: Keep the monolith, but split the workloads
#53I quite like the article and the advice it presents, building what I'd call modular monoliths (that can have modules be enabled or disabled based on feature flags) is indeed a good approach for both increasing resiliency and decreasing the blast radius of various issues. However, this bit stuck out to me: > When a bad Pub/Sub message was pulled into the binary, an unhandled panic would crash the entire app, meaning w…
Erlang and any Actors models I think
Re: Keep the monolith, but split the workloads
#54Earlier quoted context omitted.
Small note that the app in the post was not Rails, it’s a Golang app. You’re right in that PHP couldn’t fail like this though, as the isolation level is at a unix process, where the OS should contain all your errors.
The irony is that in a normal Rails app workloads are already split by default (requests and background jobs are handled by different processes that share the same codebase).
Yep, it's a very normal thing in the Rails world to separate things like this. For Django too.
I'd encourage people to look a past just the splitting of workloads though. The other point about resource limits for shared resources across workload tiers is really key, such as having very granular database connection pools even inside of a single workload deployment.
That's less common – though not unheard of – in Rails-land.
Re: Keep the monolith, but split the workloads
#55I quite like the article and the advice it presents, building what I'd call modular monoliths (that can have modules be enabled or disabled based on feature flags) is indeed a good approach for both increasing resiliency and decreasing the blast radius of various issues. However, this bit stuck out to me: > When a bad Pub/Sub message was pulled into the binary, an unhandled panic would crash the entire app, meaning w…
The most dangerous kinds of mistakes in that area are bugs or oversights that cause massive resource usage or endless loops. Those can easily bring down parts of the application that are otherwise robustly separated in the way you mentioned. Consuming too much memory will kill the app at some point, just using lots of CPU or IO will make everything else grind to a halt. Holding onto resources that are pooled like DB connections is a good way to break everything as well.
There are some languages and frameworks that are designed around even handling and isolating issues like these to some extent, Erlang/Elixir for example.
But in general I would expect a web application backend to isolate services the way you mentioned. But I don't think this is really a preconfigured default in every case, and it's not safe do catch everything and keep running. Isolating web requests and only failing the individual request is of course reasonable and should be the default. But for background services your framework can't know which ones are required and which ones are optional and when to crash the process. Especially as most languages allow shared state there, so the framework can't know how independent they are.
Re: Keep the monolith, but split the workloads
#56I quite like the article and the advice it presents, building what I'd call modular monoliths (that can have modules be enabled or disabled based on feature flags) is indeed a good approach for both increasing resiliency and decreasing the blast radius of various issues. However, this bit stuck out to me: > When a bad Pub/Sub message was pulled into the binary, an unhandled panic would crash the entire app, meaning w…
We had a single supervisor for both a Elixir/Phoenix web app and all the tasks it spawned to do long running jobs and scheduled tasks. That was very naive. A job too quickly too many times would bring down the supervisor and the web app. We moved the web app to its own supervisor tree and isolated tasks to their own supervisor too. I don't remember the details but at the end everything could crash and everything else…
That's why you use a separate `Task.Supervisor`.
Re: Keep the monolith, but split the workloads
#57Earlier quoted context omitted.
Ditto in Django + Celery
Anyone using a more modern/lightweight alternative to Celery?
Yes, dramatiq is quite good, but I ditched it for something even more lightweight -- pubsub subscribers via push notification so workers are web workers responding to http requests that might take up to 10 minutes
Re: Keep the monolith, but split the workloads
#58If I have a python monolith and I have a specific set of heavy dependency like a chromium browser, I don't want my webserver to build with it only async task. Is there a straightforward way to build a separate environment in python that just had the extra dependencies while still having benefits described in the article?
Re: Keep the monolith, but split the workloads
#59I quite like the article and the advice it presents, building what I'd call modular monoliths (that can have modules be enabled or disabled based on feature flags) is indeed a good approach for both increasing resiliency and decreasing the blast radius of various issues. However, this bit stuck out to me: > When a bad Pub/Sub message was pulled into the binary, an unhandled panic would crash the entire app, meaning w…
Have you worked with Go codebases before? Standard practice is to wrap all your entrypoints - the start of a web request, the moment you begin to run a job - in a defer recover() which will catch panics. Sadly, recover won’t apply to any subsequently goroutine’d work. That means even if your entrypoints recover, if anything go func()s down the call stack, then if that function then panics it will bring down the entir…
I probably misunderstand what you wrote. Because I think a (wrapped) panic will only result in crashing the one request that caused it?
For example Gin provides a middleware wrapper for handling panics: CustomRecoveryWithWriter returns a middleware for a given writer that recovers from any panics and calls the provided handle func to handle it.
Re: Keep the monolith, but split the workloads
#60Earlier quoted context omitted.
Have you worked with Go codebases before? Standard practice is to wrap all your entrypoints - the start of a web request, the moment you begin to run a job - in a defer recover() which will catch panics. Sadly, recover won’t apply to any subsequently goroutine’d work. That means even if your entrypoints recover, if anything go func()s down the call stack, then if that function then panics it will bring down the entir…
I probably misunderstand what you wrote. Because I think a (wrapped) panic will only result in crashing the one request that caused it? For example Gin provides a middleware wrapper for handling panics: CustomRecoveryWithWriter returns a middleware for a given writer that recovers from any panics and calls the provided handle func to handle it. ( https://github.com/gin-gonic/gin/blob/master/recovery.go )
[deleted]