Keep the monolith, but split the workloads

This is very good advice. I've been writing an article that touches on splitting workloads as part of a strategy for having an easily maintainable monolith, it goes a bit further than splitting workloads by type (worker/server/publisher) to splitting them by domain (literally redirect some endpoints to one server and others to another).

It also details some rules for restricting access between domains via clear boundaries. I should finish it and post it, I think it's a great balance between team separation and maintainability.

Do you think it's good idea to migrate from monolith to microservices before scaling the team?

We are essentially doing this right now, I personally don't think the teams are yet large enough to warrant this migration, but leadership says that we can't scale further before we have microservices. Imo most of the problems could be solved with increased testing and knowledge about the system. The tech also could definitely still scale with just by boosting the hardware and occasional performance passes on slow endpoints/queries.

I quite like the article and the advice it presents, building what I'd call modular monoliths (that can have modules be enabled or disabled based on feature flags) is indeed a good approach for both increasing resiliency and decreasing the blast radius of various issues.

However, this bit stuck out to me:

> When a bad Pub/Sub message was pulled into the binary, an unhandled panic would crash the entire app, meaning web, workers and crons all died.

I've never seen a production ready web framework or library that would let your entire process crash because of a bad request. There's always some error handling logic wrapping around the request handling, so that even if you let something slip by, the fallout will still be limited to the threads handling the requests and its context.

Now, of course there can still be greater failures, such as issues with DB access stalling for whatever reason, leading to the HTTP requests not being processed, leading to the corresponding thread pool for those filling up, as well as any request queues maxing out and thus all new requests getting rejected. But for the most part, requests with bad data and such, short of CVEs, always have limited impact.

I can imagine very few situations where things working differently wouldn't be jarring.

> but leadership says that we can't scale further before we have microservices

These are depressingly common alarm bells.

There are companies with valuations over 12 figures which don't do microservices, so the scaling justification is dubious at best.

Have you worked with Go codebases before?

Standard practice is to wrap all your entrypoints - the start of a web request, the moment you begin to run a job - in a defer recover() which will catch panics.

Sadly, recover won’t apply to any subsequently goroutine’d work. That means even if your entrypoints recover, if anything go func()s down the call stack, then if that function then panics it will bring down the entire process.

We were aware of this but ended up including one by accident anyway. It’s very sad to me that you can’t apply a global process handler to ensure this type of thing doesn’t happen, but to my knowledge that isn’t possible.

Worth mentioning Go doesn’t really encourage ‘frameworks’, and most Go apps compose together various libraries to make their app, rather than using something that packages everything together. Failures like this are an obvious downside to not having the well reviewed nuts-and-bolts-included framework to handle this for you.