Live data from Hacker News

Don't “let it crash”, let it heal

zachdaniel.dev

81–90 of 92 posts

Re: Don't “let it crash”, let it heal

#81
post #51

Earlier quoted context omitted.

I’m only an armchair expert on Erlang. But, having looked into it repeatedly for a couple decades, my take-away is the “Let it crash” slogan is good. But, also presented a bit out of context. Or, at least assuming context that most people don’t have. Erlang is used in situations involving a zillion incoming requests. If an individual request fails… Maybe it was important. Maybe it wasn’t. If it was important, it’s ex…

> You can pull this off in other languages via careful attention to the details of your request-handling code. But, the creators of the Erlang language and foundational frameworks have set their users up for success via careful attention to the design of the system as a whole. +10. So many people miss this very important point. If you have lots of mutable shared state, or can accidentally leak such into your actor co…

I have worked Elixir/Erlang and Rust a lot, and I agree. Rust in particular gives ownership semantics to threaded/blocking/locking code, which I often times find _much_ easier to understand than a series of messages sent between tasks/processes in Elixir/Erlang.

However, in a world where you have to do concurrent blocking/locking code without the help of rigorous compiler-enforced ownership semantics, Elixir/Erlang is like water in the desert.

Re: Don't “let it crash”, let it heal

#82
post #80
post #51

Earlier quoted context omitted.

> You can pull this off in other languages via careful attention to the details of your request-handling code. But, the creators of the Erlang language and foundational frameworks have set their users up for success via careful attention to the design of the system as a whole. +10. So many people miss this very important point. If you have lots of mutable shared state, or can accidentally leak such into your actor co…

The alternative to straight-line code used to be called "spaghetti code". There was a joke article parodying "GOTO considered harmful" by suggesting a "COME FROM" command. But in a lot of always, that's exactly what many modern frameworks and languages aim for.

Haha... be the change! Program in INTERCAL! :)

Re: Don't “let it crash”, let it heal

#83
I think a lot of folks who have never looked at Erlang or Elixir and BEAM before misunderstand this concept because they don't understand how fine-grained processes are, or can be, in Erlang. A very important note: Processes in BEAM languages are cheap, both to create and for context switching, compared to OS threads. While design-wise they offer similar capabilities, this cost difference results in a substantially different approach to design in Erlang than in systems where the cost of introducing and switching between threads is more expensive.

In a more conventional language where concurrency is relatively expensive, and assuming you're not an idiot who writes 1-10k SLOC functions, you end up with functions that have a "single responsibility" (maybe not actually a single responsibility, but closer to it than having 100 duties in one function) near the bottom of your call tree, but they all exist in one thread of execution. In a system, hypothetical, created in this model if your lowest level function is something like:

  retrieve_data(db_connection, query_parameters) -> data
And the database connection fails, would you attempt to restart the database connection in this function? Maybe, but that'd be bad design. You'd most likely raise an exception or change the signature so you could express an error return, in Rust and similar it would become something like:

  retrieve_data(db_connection, query_parameters) -> Result
Somewhere higher in the call stack you have a handler which will catch the exception or process the error and determine what to do. That is, the function `retrieve_data` crashes, it fails to achieve its objective and does not attempt any corrective action (beyond maybe a few retries in case the error is transient).

In Erlang, you have a supervision tree which corresponds to this call tree concept but for processes. The process handling data retrieval, having been given some db_conn handler and the parameters, will fail for some reason. Instead of handling the error in this process, the process crashes. The failure condition is passed to the supervisor which may or may not have a handler for this situation.

You might put the simple retry policy in the supervisor (that basic assumption of transient errors, maybe a second or third attempt will succeed). It might have other retry policies, like trying the request again but with a different db_connection (that other one must be bad for some reason, perhaps the db instance it references is down). If it continues to fail, then this supervisor will either handle the error some other way (signaling to another process that the db is down, fix it or tell the supervisor what to do) or perhaps crash itself. This repeats all the way up the supervision tree, ultimately it could mean bringing down the whole system if the error propagates to a high enough level.

This is conceptually no different than how errors and exceptions are handled in sequential, non-concurrent systems. You have handlers that provide mechanisms for retrying or dealing with the errors, and if you don't the error is propagated up (hopefully you don't continue running in a known-bad state) until it is handled or the program crashes entirely.

In languages that offer more expensive concurrency (traditional OS threads), the cost of concurrency (in memory and time) means you end up with a policy that sits somewhere between Erlang's and a straight-line sequential program. Your threads will be larger than Erlang processes so they'll include more error handling within themselves, but ultimately they can still fail and you'll have a supervisor of some sort that determines what happens next (hopefully).

As more languages move to cheap concurrency (Go's goroutines, Java's virtual threads), system designs have a chance to shift closer to Erlang than that straight-line sequential approach if people are willing to take advantage of it.

Re: Don't “let it crash”, let it heal

#84

It is very strange that a post trying to explain the concept of "let it crash" in Elixir (which runs on the BEAM VM) does not mention the doctoral thesis of Joe Armstrong: "Making reliable distributed systems in the presence of software errors". It must be compulsory lecture for anybody interested in reliable systems, even if they do not use the BEAM VM. https://www.diva-portal.org/smash/record.jsf?pid=diva2%3A104...

[deleted]

Re: Don't “let it crash”, let it heal

#85

It is very strange that a post trying to explain the concept of "let it crash" in Elixir (which runs on the BEAM VM) does not mention the doctoral thesis of Joe Armstrong: "Making reliable distributed systems in the presence of software errors". It must be compulsory lecture for anybody interested in reliable systems, even if they do not use the BEAM VM. https://www.diva-portal.org/smash/record.jsf?pid=diva2%3A104...

Some core ideas from the paper for the inpatient (failures, isolation, healing):

- Failures are inevitabe, so systems must be designed to EXPECT and recover from them, NOT AVOID them completely.

- Let it crash philosophy allows components to FAIL and RECOVER quickly using supervision trees.

- Processes should be ISOLATED and communicate via MESSAGE PASSING, which prevents cascading failures.

- Supervision trees monitor other processes and RESTART them when they fail, creating a self-healing architecture.

Re: Don't “let it crash”, let it heal

#86
post #46

Earlier quoted context omitted.

In general, if you can move any kind of logic to a lower level, that's better. For example, testing that kubernetes restarts work correctly is tricky and requires a complicated setup. Testing that an erlang process/actor behaves as expected is basically a unit test.

I bet the kubernetes project has test for that, why should I as an application developer care about testing something other than my own code?

That's assuming your code is well-configured. How do you test your k8s configs?

Re: Don't “let it crash”, let it heal

#87
post #58

Earlier quoted context omitted.

> in which case the whole program may finally crash. This may happen if you let it, but it's basically never the desired outcome. If you were handling a user request, it should stop by returning a HTTP 500 to the client, or if you were processing a background job of some sort, it should stop with a watchdog process marking the job as a failure, not with the entire system crashing.

returning HTTP 500 as early as possible is an example of "let it crash" approach outside of Erlang.

That's not what "let it crash" is about. Letting something crash in Erlang means that a process (actor) is allowed to crash, but then it gets restarted to try again, which would resolve the situation in case of transient errors.

The equivalent of "let it crash" outside of Erlang is a mountain of try-catch statements and hand-rolled retry wrappers with time delays, with none of the observability and tooling that you get in Erlang.

Re: Don't “let it crash”, let it heal

#88

Ah this makes sense. I always thought "let it crash" made it sound like Elixir devs just don't bother with error checking, like writing Java without any `catch`es, or writing Rust that only uses `.unwrap()`. If they just mean "processes should be restartable" then that sounds way more reasonable. Similar idea to this but less fancy: https://flawless.dev/ It's a pretty terrible slogan if it makes your language sound w…

Flawless is interesting.

It can't work in the general case because replaying a sequence of syscalls is not sufficient to put the machine back in the same state as it was last time. E.g. second time around open behaves differently so you need to follow the error handling.

However sometimes that approach would work. I wonder how wide the area of effective application is. It might be wide enough to be very useful. The all or nothing database transaction model fits it well.

Re: Don't “let it crash”, let it heal

#89

Earlier quoted context omitted.

Elixir dev: It does not solve all issues. But sometimes you have some kind of rare bug that just happens once X,Z and Y happens in a specific order. If it is restarted it might not happen that way again. Or it might be a temporary problem. You are reaching for an API and it temporarily has issues. It might not have it anymore in 50 ms. But of course if it crashes because you are reading a file that does not exist it…

Note that let is crash doesnt mean we shouldnt fix bugs. It is more about if there is a bug we havent fixed it is better to make the crash just crash a tiny part of the program than the whole program

Or more importantly, you can't design robust recovery and retry systems.

Re: Don't “let it crash”, let it heal

#90

https://fsharpforfunandprofit.com/rop/ Railway orientated programming to the rescue?

There are a couple of patterns for accomplishing this in Elixir.

One is to build multiple function heads that pattern match on the arguments. If it’s an error tuple, pass it along. Build up your pipeline and handle any errors at the end.

Another is to use the `with else`[0] expression for building up a railroad. This has the benefit of not having to teach your functions how to pass along errors. Error handling in the else block can be a little gnarly.

I find it a little more manual than languages that have a `runEffect` or compose operator. In large part that’s due to the :ok, :error tuples being more of a convention than a primitive like Either/Result.

0: https://elixirschool.com/en/lessons/basics/control_structure...

Post reply on HN