How “let it fail” leads to simpler code

Other frameworks like express (nodejs) or actix (Rust) also don't crash if you "throw" in an request handler so this doesn't sounds very exciting to me. The interesting question for me is how retries are handled after an error occurred? For example, if the error happens in an http request handler, does the request still fails with 500 or is it magically retried by Erlang while keeping the request hanging? For internal service calls how are retries working? i.e. how can I configure that a request is retried after a failure? I guess Erlang does this and this is the power behind it?

The example of a missing file seems not very good since its a problem that is probably not solved by waiting. A better example is probably a busy DB that is temporary not reachable?

My guess would be add assertions everywhere instead of throwing exceptions.

Totally agree. Now, I’ve actually found functional programming - specifically, Either - to be of great use in helping me focus only on the happy path.

These kind of systems are not always appropriate, but when they do, they work wonderfully.

Our use case was to build a service to manage the dynamic part of our infrastructure. These are infra pieces that are created/deleted/modified on the fly according to some policies, instead of defining them statically as code. The implementation is simply a lambda function that runs every minute, loads a policy, compares the current state of the system, and then creates/deletes/modify resources as needed.

I am currently in the process of writing a talk that I will deliver to the rest of my org. This will help me crystalize my thoughts, but here are some pointer on why I think it worked in this case:

* The service is stateless. On each run it just loads a policy, compares it to the current state of the system, and acts accordingly. This avoids handling complicated state or coordinating executions. In theory two policies could contradict each other, but in practice we partition our policies in such a way overlap is not possible.

* Operations are idempotent. This is one of the reasons the system converges to a desired state. This makes the service resilient to both failures and eventual consistency.

* Deviation from policies doesn't affect correctness. We are fortunate that our system is not directly customer facing. Deviation from policies affects only performance. The system can work several minutes (if not hours) outside policy band without consequences. This probably will be a critical blocker for most production systems.

Other than control theory and let it fail, I had the chance to play with other cool concepts while implementing this service. These are some of those:

* Parse, don't validate/anti-corruption layer: The service downloads and parses a policy at the beginning of each run. If parsing fails, it errors out. Otherwise, it passes the policy object to the rest of the execution. This makes the system easy to test, and avoids the anti-pattern of peppering your code with instructions reading input, just to find mid-execution that the policy was invalid.

* Pluggable policies: The main body of the service is a very simple sense/act loop. For the actors, we use a strategy pattern, where policies can choose what strategy to use. This approach has helped us to introduce new behavior with minimal code changes.

* Typescript as configuration language: This service replaces an older, less flexible one. A major pain of the old service is that policies where defined as Jinja templates over plain text files. This became unmaintainable as the number and complexity of policies grew. Our new service defines policies in Typescript. Policies are statically typed, and we use regular programming constructs (functions, loops, variables...) to build them at compile time. The output is still a plain JSON file.

Hope that helps.

>ignore unexpected exceptions

Isn't it the way it already is in practice, not something specific to Erlang? If an exception is unexpected, usually there won't be an exception handler for it, otherwise a developer pretty much expected it. Developers are generally lazy so in my practice the default is usually to let it fail, and there's usually going to be an exception handler that does something other than logging and quitting only if there's a serious reason to do so.

Maybe a more useful distinction could rather be "business logic errors" vs. everything else ("infrastructure errors", "programming errors" and "input validation errors"). Business logic should clearly define what should be done when an error happens, to avoid inconsistent state. But infrastructure-level errors or programming errors, you can't do much about them, other than log and/or retry.