Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Keep the monolith, but split the workloads

incident.io

Post reply on HN
Print view View original

151–160 of 161 posts

Re: Keep the monolith, but split the workloads

#151

Post by withinboredom »

dontlaugh wrote: post #147
withinboredom wrote: post #140

Earlier quoted context omitted.

Aren’t you doing that anyway with json, protobufs, or whatever and using RPC? There’s always some agreed upon contract between services and changing that contract can always be delicate.

If you depend both on schema and RPC, it becomes very difficult to gradually deploy changes without downtime. You’re forced to be both forwards and backwards compatible. Worse, schemas are often only part of the contract. It’s very common to have logic and constraints in code, maybe even a majority. To share that across services you end up with shared models, then you end up with some utils with logic, before you kno…

Ah, I agree with your conclusion but not about sharing db's. Sometimes its literally the only way to share data (particularly very large sets of data).

Re: Keep the monolith, but split the workloads

#152

Post by tkahnoski »

KronisLV wrote: post #36

I quite like the article and the advice it presents, building what I'd call modular monoliths (that can have modules be enabled or disabled based on feature flags) is indeed a good approach for both increasing resiliency and decreasing the blast radius of various issues. However, this bit stuck out to me: > When a bad Pub/Sub message was pulled into the binary, an unhandled panic would crash the entire app, meaning w…

Depends on the underlying framework on how it isolates processes... I could see this happening in a monolithic JVM application where it pretends there are separate containers, but a fatal error in the JVM will crash the world on that server.

A related example I lived through with a Perl application, someone decided to use this library 'Storable' that would serialize the memory in a binary format. We upgraded the library and started seeing "slow" performance across the server farm.

We recognized the processes were intermittently crashing and after decoding core dumps... figured out it was this upgrade to the Storable library. Apache httpd server chugged along just fine restarting processes. So different run-time, different type of crash resiliency.

Long-term lesson... be extra cautious with memory-serialized objects. Newer libraries have better protection on this to parse a header to detect compatible issues before loading the raw object into memory, but the potential is there especially with distributed systems today.

Re: Keep the monolith, but split the workloads

#153

Post by brentjanderson »

cle wrote: post #148
brentjanderson wrote: post #68

Earlier quoted context omitted.

Erlang/Elixir have a great story here: “let it crash”. Because each slice of activity in an application is wrapped in its own process (think single threaded loop but you can run a million at a time, almost free to create and destroy), if it crashes it only takes down that web request/process. Recovery mechanisms are built in to get back to a know good state.

I haven't used Erlang extensively, what happens if you crash in the middle of e.g. holding a lock, or during a coordinated dance with other processes? My concern isn't really "does the program keep running?", it's "does the program keep running correctly ?".

That sort of problem is beyond the scope of the runtime in any case, isn't it? In either of the examples you offered (holding a lock, coordinating with other processes), there must be timeouts enforced by the lock or the other processes so that, if something goes wrong, the system isn't waiting for a crashed process to continue the work.

Erlang/Elixir do make this pretty easy to manage, including the scenario where the process does recover by reverting back to a known good state. It won't do it for you automatically, but it exposes enough surface area to make problems like that solvable without reaching for a lot of extra tools - it's built into the runtime.

Re: Keep the monolith, but split the workloads

#154

Post by dontlaugh »

withinboredom wrote: post #151
dontlaugh wrote: post #147

Earlier quoted context omitted.

If you depend both on schema and RPC, it becomes very difficult to gradually deploy changes without downtime. You’re forced to be both forwards and backwards compatible. Worse, schemas are often only part of the contract. It’s very common to have logic and constraints in code, maybe even a majority. To share that across services you end up with shared models, then you end up with some utils with logic, before you kno…

Ah, I agree with your conclusion but not about sharing db's. Sometimes its literally the only way to share data (particularly very large sets of data).

I generally think micro services are a bad idea, partly because it’s so useful to share data and code implicitly.

Somewhat more macro services can make sense, especially when each team owns a service. That’s also when sharing data is harder.

Re: Keep the monolith, but split the workloads

#155

Post by paulddraper »

cle wrote: post #64
lawrjone wrote: post #40

Earlier quoted context omitted.

Have you worked with Go codebases before? Standard practice is to wrap all your entrypoints - the start of a web request, the moment you begin to run a job - in a defer recover() which will catch panics. Sadly, recover won’t apply to any subsequently goroutine’d work. That means even if your entrypoints recover, if anything go func()s down the call stack, then if that function then panics it will bring down the entir…

To me, it's unclear what the best solution is here. Other languages solve this differently with tradeoffs, e.g. Java defaults to threads silently dying when an exception isn't caught. Your program will continue to run, but it's probably in some undefined state at that point. There are mechanisms for propagating exceptions elsewhere, but they have to be explicitly set up (like in Go). You can set a default uncaught ex…

Node.js changed behaviors over time.

Re: Keep the monolith, but split the workloads

#156

Post by gmokki »

dontlaugh wrote: post #147
withinboredom wrote: post #140

Earlier quoted context omitted.

Aren’t you doing that anyway with json, protobufs, or whatever and using RPC? There’s always some agreed upon contract between services and changing that contract can always be delicate.

If you depend both on schema and RPC, it becomes very difficult to gradually deploy changes without downtime. You’re forced to be both forwards and backwards compatible. Worse, schemas are often only part of the contract. It’s very common to have logic and constraints in code, maybe even a majority. To share that across services you end up with shared models, then you end up with some utils with logic, before you kno…

We had microservices that shared a database, but each got a separate schema. We decided to treat cross schema queries the same as rpc: we export and API from a schema (view or stored procedure) that other microservices can depend on.

The microservice that exposes database level API can do any updates as long as the API stays the same (exactly same as with classical rpc).

Re: Keep the monolith, but split the workloads

#157

Post by withinboredom »

dontlaugh wrote: post #154
withinboredom wrote: post #151

Earlier quoted context omitted.

Ah, I agree with your conclusion but not about sharing db's. Sometimes its literally the only way to share data (particularly very large sets of data).

I generally think micro services are a bad idea, partly because it’s so useful to share data and code implicitly. Somewhat more macro services can make sense, especially when each team owns a service. That’s also when sharing data is harder.

Yep. The only example I have from my career was a team having their own read replica and ETLing that to Hadoop. The schema hadn’t changed in 10+ years, so it wasn’t even considered a real risk.

Re: Keep the monolith, but split the workloads

#158

Post by Arbortheus »

KronisLV wrote: post #36

I quite like the article and the advice it presents, building what I'd call modular monoliths (that can have modules be enabled or disabled based on feature flags) is indeed a good approach for both increasing resiliency and decreasing the blast radius of various issues. However, this bit stuck out to me: > When a bad Pub/Sub message was pulled into the binary, an unhandled panic would crash the entire app, meaning w…

My team has experienced _exactly_ the same cyclical panicking in production before because of the same library.

The Google Pub/Sub Go library does not handle panics by default, so if any message payload plays badly with your code and panics, you cyclically panic the service.

That's because the message keeps getting retried because you don't `Ack` it. Non acknowledgements get retried automatically, you get the picture.

Re: Keep the monolith, but split the workloads

#159

Post by Arbortheus »

vivegi wrote: post #145

Leave the Gun, Take the Cannoli > When a bad Pub/Sub message was pulled into the binary, an unhandled panic would crash the entire app, meaning web, workers and crons all died. Several years ago while working on a Message Queue based application, we had a poison message , one where the consumer process would read the message and then die because the message was malformed and then the message would get back to the que…

I suppose PubSub is at least slightly better than this because it's not a perfect queue. I.E. messages that are not acknowledged will be retried with exponential backoff.

That at least gives some time to process some stuff before you encounter the poison message again.

Whereas with a FIFO queue you're completely screwed.

Re: Keep the monolith, but split the workloads

#160

Post by funcDropShadow »

dboreham wrote: post #99
cle wrote: post #64

Earlier quoted context omitted.

To me, it's unclear what the best solution is here. Other languages solve this differently with tradeoffs, e.g. Java defaults to threads silently dying when an exception isn't caught. Your program will continue to run, but it's probably in some undefined state at that point. There are mechanisms for propagating exceptions elsewhere, but they have to be explicitly set up (like in Go). You can set a default uncaught ex…

> when an exception isn't caught Not catching all exceptions is a glaring P0 bug.

You should almost never catch all exceptions, i.e. Throwable on the JVM. That is one of the few things that Scala really got right. The `catch NonFatal(e) =>` idiom is doing that nicely. It will catch all throwables with a selected set of special cases, e.g. OutOfMemoryException and all the other VirtulMachineErrors. Catching those in a framework lead to extending the time until a crash follows on a serious issue. Crashing early is often beneficial in such a situation. Together with a process watchdog, like systemd, kubernetes, dockerd, whatever crashing early increases the uptime.
Post reply on HN