Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Keep the monolith, but split the workloads

incident.io

Post reply on HN
Print view View original

111–120 of 161 posts

Re: Keep the monolith, but split the workloads

#111

Post by hardware2win »

ohgodplsno wrote: post #97
hardware2win wrote: post #81

Earlier quoted context omitted.

But if 1 database dies, then all microservices dont work So why are you even using them? The benefit of microservices is also reliability, which you just threw away Isnt this basically distributed monolith? So you combined bad things of both worlds! Single point of failure of monolith And deployment difficulties and need for saga like patterns to deal with network trickiness from distributed designs Whats the point?

Once again, cargo-cultish reasoning, where it's all or nothing. 1/ Replication & read-only copies for resilience exist, to allow you to function in degraded state if your database goes down. 2/ If the database for one service goes down, any service that calls it ends up being down anyways. No, being able to respond with a 503 that says that megatron-service is down and you can't fetch the data isn't different from re…

1st point is really good, make sense!

Your 3rd point shows scenerio where 2nd point is not applicable.

>what kind of fucking code have you seen that a whole server will not start because it can't connect to a database ?

Server? Idk. App? Sure, CRUD app that requires config/filling cache dictionaries from db

>Once again, cargo-cultish reasoning, where it's all or nothing.

Thats fair point, so what are micro services doing for you then?

Allowing teams to deploy independently?

Re: Keep the monolith, but split the workloads

#112

Post by vsareto »

hardware2win wrote: post #81
ohgodplsno wrote: post #77

Earlier quoted context omitted.

If your database isn't the performance issue and those two services spent 99% of the response time on calculating the 15755th digit of pi, not separating the database isn't a problem. Similarly, if both of your services need to fetch some user information, but you haven't created a user information service with its own database they could both talk to, having a shared database is fine. Splitting each and every servic…

But if 1 database dies, then all microservices dont work So why are you even using them? The benefit of microservices is also reliability, which you just threw away Isnt this basically distributed monolith? So you combined bad things of both worlds! Single point of failure of monolith And deployment difficulties and need for saga like patterns to deal with network trickiness from distributed designs Whats the point?

It wouldn't be unreasonable to say database products are an order of magnitude or two more reliable than code written by many teams.

There's some advantage in having code separation first, but it is admittedly not a benefit of reliability.

These half-measures are generally done when converting an older architecture to a newer one. I don't think many places actually do big bang conversions/releases going from monolith -> microservices where you can do everything correctly. That runs counter to the CI/CD trends that we taught them.

The other issue might be a conflict in database table constraints (or worse, triggers/sprocs) that don't match your microservice boundaries, so the business ends up wanting wins faster than your team can resolve those conflicts.

Re: Keep the monolith, but split the workloads

#113

Post by tough »

jameshart wrote: post #105
lawrjone wrote: post #101

Earlier quoted context omitted.

It’s a Go app, not a Ruby one. Hence the Go code examples. And the rationale behind one binary rather than several is to reduce build time and deployment artefact size, along with benefits in development from one binary. All of which mean deploying is much quicker/easier, which means it’s less of a big deal that we deploy everything whenever any part changes (we deploy anywhere up to 30 times a day). Hope that helps…

Apologies - read the ‘Ruby monolith’ up top and then glossed over the code without noticing the language mismatch. Mondays, am I right? I don’t understand how bundling three sets of functionality into one binary reduces build time or artifact size. Surely the binary containing all the web and pubsub and scheduled tasks is strictly bigger than the binary for just the web would be, and takes longer to build and test th…

Maybe the three binaries have shared deps, by bundling them on one you get to not duplicate those shared deps?

I dunno as I have never used Go, but this would make sense with node

Also thinking how all the related tasks a la CI/CD might be total time faster with one process than three, since they're doing less total work, both during tasks and to wake_up the process etc dunno just speculation on my part

Re: Keep the monolith, but split the workloads

#114

Post by ohgodplsno »

hardware2win wrote: post #111
ohgodplsno wrote: post #97

Earlier quoted context omitted.

Once again, cargo-cultish reasoning, where it's all or nothing. 1/ Replication & read-only copies for resilience exist, to allow you to function in degraded state if your database goes down. 2/ If the database for one service goes down, any service that calls it ends up being down anyways. No, being able to respond with a 503 that says that megatron-service is down and you can't fetch the data isn't different from re…

1st point is really good, make sense! Your 3rd point shows scenerio where 2nd point is not applicable. >what kind of fucking code have you seen that a whole server will not start because it can't connect to a database ? Server? Idk. App? Sure, CRUD app that requires config/filling cache dictionaries from db >Once again, cargo-cultish reasoning, where it's all or nothing. Thats fair point, so what are micro services d…

The third point is directly linked to the second point: If you are doing a call that requires access to that database, whether it's going through two services or one, will lead to an error (just a different one). If you are doing a call that doesn't require it, it'll go through smoothly.

>Server? Idk. App? Sure, CRUD app that requires config/filling cache dictionaries from db

Not once in my life have I written either a server or an app that will completely fail upon not having a database (the .php files written in early college years don't count.). Either of these can function in a degraded way without the database; In the case of an app, still display your UI, still display everything that can be displayed without the presence of the database, and warn that said database is down. The same way you'd do with micro services.

>Thats fair point, so what are micro services doing for you then? >Allowing teams to deploy independently?

Where I currently work, our services are gathered in various bills of materials to have a set of components that we know works, and that the client requires (government work is always fun). Multiply this by 20 different environments, and you're quite happy to have those BoMs. However, it could very much be done without microservices. Microservices allow us to bypass some problems: One part of the server really, really, really need to read a lot of data and to build a graph before starting, which can in some cases take upwards of two hours. Anything that doesn't depend on this would be very happy to start up without waiting for that to happen, so this is when you split it.

Don't do microservices because it's the hip thing to do. Micro services are just one more tool to alleviate large problems. Sometimes, splitting it into two services is the right thing to do. Sometimes, sharing the database is the right thing to do.

Re: Keep the monolith, but split the workloads

#115

Post by lawrjone »

jameshart wrote: post #105
lawrjone wrote: post #101

Earlier quoted context omitted.

It’s a Go app, not a Ruby one. Hence the Go code examples. And the rationale behind one binary rather than several is to reduce build time and deployment artefact size, along with benefits in development from one binary. All of which mean deploying is much quicker/easier, which means it’s less of a big deal that we deploy everything whenever any part changes (we deploy anywhere up to 30 times a day). Hope that helps…

Apologies - read the ‘Ruby monolith’ up top and then glossed over the code without noticing the language mismatch. Mondays, am I right? I don’t understand how bundling three sets of functionality into one binary reduces build time or artifact size. Surely the binary containing all the web and pubsub and scheduled tasks is strictly bigger than the binary for just the web would be, and takes longer to build and test th…

No problem, easy to skim by it!

In terms of how this reduces artifact size: Go statically compiles everything, and the majority of that weight is from dependencies. Compiling into three different binaries would:

1. Increase build time, because even while the dependencies are cached from each subsequent build, linking isn't free. Takes about 20s to link each binary, so it adds about 40s+ to the build to create them twice.

2. Increases build size, because each binary is about 150MB, so we go from 150 -> 450MB, which increases time to upload to container registry and download into the infrastructure that runs things.

In answer to the security surface, the code between cron/worker/web is very interlinked. There would be little practical reduction in surface area from splitting the binary, though perhaps for some applications there might.

Re: Keep the monolith, but split the workloads

#116

Post by rfoo »

lawrjone wrote: post #50
rfoo wrote: post #29

Earlier quoted context omitted.

Question: Have you considered one separate binary (but still sharing the entire codebase) for each "workload" instead of using a flag to switch? If so, can you highlight the advantage of the flag way?

Yep: this is definitely a valid alternative. The reasons we went with one binary are: - We can ensure common runtime setup by only having one codepath that runs the initialisation. It's nice that every deployment definitely instantiates the Prometheus server and initialises secrets from Google Secret Manager at the same point and in the same way. - Compiling three targets adds time to our build process. Even when Go…

Thanks!

Re: Keep the monolith, but split the workloads

#118

Post by kubanczyk »

lawrjone wrote: post #80
zimpenfish wrote: post #71

Earlier quoted context omitted.

> Have you worked with Go codebases before? Several but ... > Standard practice is to wrap all your entrypoints [...] in a defer recover() ... I've never seen that. Is there some literature pointing to this as best practice?

You’ll find some libraries do this for you, such as HTTP servers. They do this because if your server code makes a mistake such as accessing a nil pointer, a segfault or panic would bring down the entire process. That’s why you want to recover(), to avoid a your process dying.

> They do this because if your server code makes a mistake

This is neither good or best practice.

My take:

- Know that there are simple-mistake panics, and internal-state-just-went-bonkers panics. For the latter, you can guess the boundary of impact (one request, one connection, one job, one userID, one process, ...) but exit(1) is much more reliable than guessing.

- Tests can easily catch simple mistakes like accessing a nil pointer.

- Know that the kind of programmer who does not care about simple tests, will also not care about concurrency bugs which introduce the more dangerous types of state corruption. This corruption would likely be not limited to a single request.

Good luck to frameworks who assume that nothing bad would happen if they ignore a panic and continue serving more requests.

Re: Keep the monolith, but split the workloads

#119

Post by jrockway »

lawrjone wrote: post #80
zimpenfish wrote: post #71

Earlier quoted context omitted.

> Have you worked with Go codebases before? Several but ... > Standard practice is to wrap all your entrypoints [...] in a defer recover() ... I've never seen that. Is there some literature pointing to this as best practice?

You’ll find some libraries do this for you, such as HTTP servers. They do this because if your server code makes a mistake such as accessing a nil pointer, a segfault or panic would bring down the entire process. That’s why you want to recover(), to avoid a your process dying.

I mean, net/http does it. That's the standard library.

A go convention that I've made up, or that is perhaps a real one, is to always look at the standard library for guidance on how to write Go. net/http suffers a little bit from being a very early library and thus you might not want to emulate its API surface. But in general, the Go team thought "you know what every HTTP handler in Go needs? recovery from panics" and that is worth some weight when considering your own design.

I would personally recommend fuzz testing your code, including HTTP handlers. The more panics you find in development, the fewer customers you lose from panics in production. Remember that recovering from a panic in an HTTP handler still means your user's request didn't get processed. They are not happy about that, even if your program can still service other users.

Re: Keep the monolith, but split the workloads

#120

Post by dontlaugh »

ohgodplsno wrote: post #77
hardware2win wrote: post #66

Earlier quoted context omitted.

>extreme microservices >separate databases Isnt it like the basic thing?

If your database isn't the performance issue and those two services spent 99% of the response time on calculating the 15755th digit of pi, not separating the database isn't a problem. Similarly, if both of your services need to fetch some user information, but you haven't created a user information service with its own database they could both talk to, having a shared database is fine. Splitting each and every servic…

Why even bother then? If your code shares the same database then it’s a single service. Anything else is just unnecessary overhead.
Post reply on HN