Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Keep the monolith, but split the workloads

incident.io

Post reply on HN
Print view View original

81–90 of 161 posts

Re: Keep the monolith, but split the workloads

#81

Post by hardware2win »

ohgodplsno wrote: post #77
hardware2win wrote: post #66

Earlier quoted context omitted.

>extreme microservices >separate databases Isnt it like the basic thing?

If your database isn't the performance issue and those two services spent 99% of the response time on calculating the 15755th digit of pi, not separating the database isn't a problem. Similarly, if both of your services need to fetch some user information, but you haven't created a user information service with its own database they could both talk to, having a shared database is fine. Splitting each and every servic…

But if 1 database dies, then all microservices dont work

So why are you even using them?

The benefit of microservices is also reliability, which you just threw away

Isnt this basically distributed monolith?

So you combined bad things of both worlds!

Single point of failure of monolith

And deployment difficulties and need for saga like patterns to deal with network trickiness from distributed designs

Whats the point?

Re: Keep the monolith, but split the workloads

#82

Post by maeln »

EGreg wrote: post #53
KronisLV wrote: post #36

I quite like the article and the advice it presents, building what I'd call modular monoliths (that can have modules be enabled or disabled based on feature flags) is indeed a good approach for both increasing resiliency and decreasing the blast radius of various issues. However, this bit stuck out to me: > When a bad Pub/Sub message was pulled into the binary, an unhandled panic would crash the entire app, meaning w…

Erlang and any Actors models I think

Yes and no. Erlang does apply the "fail fast" idea and recommend to let actor crash instead of trying to handle the error (with some pragmatism ofc). But we are not talking about crashing the process of the whole app. In most case in Erlang (but you find the same idea with over massively parallel system or with micro-services), your actor will be handled by a supervisor, which job it is to keep it alive, and so restart it when it crash. Also, most of the time in Erlang, having a actor crash and spawning a new one is really fast, so you can afford to crash even for something as basic as a malformed json request or similar. Not something you can really afford when spawning a new system is more in the order of several hundred ms.

Re: Keep the monolith, but split the workloads

#83

Post by mjarrett »

> No complex dev setup required, or RPC framework needed, it’s the same old monolith, just operated differently.

It seems to me like "operated differently" is doing a lot of heavy lifting that often involves those same frameworks or dev/testing environments. If a monolith used to communicate between workloads in-process, now there needs to be some way for those workloads to communicate between processes, and it has to continue to work in dev. The example in the article mentions roundtripping through something like Redis or Postgres, but now doesn't your dev environment need to spin up a database? What if the communication pattern isn't a shared cache, but instead a direct function call that, say, spins up a new goroutine to process some work asynchronously? Now you need to operate and maintain either an RPC protocol or an event queue, and make sure those things can be started up and discoverable during testing and local development.

Re: Keep the monolith, but split the workloads

#84

Post by Gys »

lawrjone wrote: post #62
Gys wrote: post #59

Earlier quoted context omitted.

I probably misunderstand what you wrote. Because I think a (wrapped) panic will only result in crashing the one request that caused it? For example Gin provides a middleware wrapper for handling panics: CustomRecoveryWithWriter returns a middleware for a given writer that recovers from any panics and calls the provided handle func to handle it. ( https://github.com/gin-gonic/gin/blob/master/recovery.go )

The post-mortem we published about this outage explains the details of how this crashed the binary. It gives some code examples and explanations that should clear this up: https://incident.io/blog/intermittent-downtime#mitigation-1-... ^ link should go to the relevant section

Will have a look, thanks!

Re: Keep the monolith, but split the workloads

#85

Post by mattmanser »

mjarrett wrote: post #83

> No complex dev setup required, or RPC framework needed, it’s the same old monolith, just operated differently. It seems to me like "operated differently" is doing a lot of heavy lifting that often involves those same frameworks or dev/testing environments. If a monolith used to communicate between workloads in-process, now there needs to be some way for those workloads to communicate between processes, and it has t…

It's pretty normal to have a local dev database on your machine. It's how we did it for decades. Are there really developers in the wild now that have never been exposed to working with a locally installed SQL instance?

These days you can even set it up in docker, automatically migrate the database to the latest version and install a bunch of test data. So you can wipe the whole thing and go back to a good state if you muck it up with a bad or ill-thought-out local migration, etc.

Same with redis, etc.

And it's still much, much simpler than a microservice architecture.

Re: Keep the monolith, but split the workloads

#86

Post by nickdandakis »

mjarrett wrote: post #83

> No complex dev setup required, or RPC framework needed, it’s the same old monolith, just operated differently. It seems to me like "operated differently" is doing a lot of heavy lifting that often involves those same frameworks or dev/testing environments. If a monolith used to communicate between workloads in-process, now there needs to be some way for those workloads to communicate between processes, and it has t…

You should already be spinning up caches and databases in dev anyway?

I agree though that the article is missing some explicit insight into how this change is handled on the local dev environment. I'm assuming the local dev environment run commands were also updated to be these three commands, one per workload.

Basically, this distinction should be represented throughout all environments, dev/test/prod

Re: Keep the monolith, but split the workloads

#87

Post by exabrial »

Current job we have a rule: we should not write no more than four layers deep of abstraction in our codebase. The thought being any more that is too much for more than a single person to keep in their head.

I feel like this slays the monolith and forces developers to really think out stuff before they write it. There is of course exceptions to everything including this, but it’s served us pretty well.

Re: Keep the monolith, but split the workloads

#89

Post by svieira »

exabrial wrote: post #87

Current job we have a rule: we should not write no more than four layers deep of abstraction in our codebase. The thought being any more that is too much for more than a single person to keep in their head. I feel like this slays the monolith and forces developers to really think out stuff before they write it. There is of course exceptions to everything including this, but it’s served us pretty well.

I'm curious as to how this plays out in practice (though I love the theory). Some questions:

1. Is this measured in _depth_, in breadth, or in both directions (and if both is the total calculated with `MAX` or `SUM`)?

2. Do you run multiple `main`s, each with this restriction?

3. Do you count third-party frameworks and libraries as layers of abstraction?

Re: Keep the monolith, but split the workloads

#90

Post by wruza »

gbuk2013 wrote: post #48
Aeolun wrote: post #47

Earlier quoted context omitted.

> I've never seen a production ready web framework or library that would let your entire process crash because of a bad request. Node does this (regardless of framework/server) if you have some unhandled promise rejection/error in an asynchronous function.

You can catch these with Node but I wouldn’t do it - if it’s not explicitly handled then it’s either a bug or something really bad happened and the app needs to get alarm bells ringing.

Error recovery in long-running dynamic-y processes is nontrivial, especially for “programming errors”. While it can be done, much easier to stick to crashonly practices. Also helps in production where you can safely crash it and restart without worrying about stale something.
Post reply on HN