Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How “let it fail” leads to simpler code

yiming.dev

Post reply on HN
Print view View original

101–110 of 165 posts

Re: How “let it fail” leads to simpler code

#101

Post by tsimionescu »

angarg12 wrote: post #98
tsimionescu wrote: post #82

Earlier quoted context omitted.

This is exactly why I think all the discussions about the importance of error handling paths (and the aversion drive have to exceptions) are usually overblown. The most successful, and common, error handling strategy is to log and abandon the whole operation, cleaning up everything the operation left around. If you have one process per operation, this is often very well captured by doing exit() at the place of the er…

Funny that you mention cleanup. Our service doesn't clean up in any circumstance. Originally we had a cleanup operation in case of errors. But then we found those could fail as well. As it turns out, we need a catch-all way to clean up resources if anything else fails. The solution is simple. The main service never cleans up, and a secondary service (I like to call it The Reaper) just cleans up orphaned resources. Th…

Sure, if you can avoid cleaning up, that's great. The biggest potential problem I know of are client-side TCP connections, which aren't very easy to clean up after the fact (you'd have to know the client port, and craft a RST packet from that port), and tend to have really long timeouts by default.

Re: How “let it fail” leads to simpler code

#102

Post by tsimionescu »

dfee wrote: post #95
tsimionescu wrote: post #82

Earlier quoted context omitted.

This is exactly why I think all the discussions about the importance of error handling paths (and the aversion drive have to exceptions) are usually overblown. The most successful, and common, error handling strategy is to log and abandon the whole operation, cleaning up everything the operation left around. If you have one process per operation, this is often very well captured by doing exit() at the place of the er…

Totally agree. Now, I’ve actually found functional programming - specifically, Either - to be of great use in helping me focus only on the happy path.

Yes, with some kind of monadic-like control flow (either actual monads or even Rust's ? macro), those can be also achieve this workflow pretty well.

Edit to add: I still think exceptions are better in practice, as you also get a stack trace when the failure happens, whereas Either and ? don't really help track down the error unless you add code to create a manual "stack trace".

Re: How “let it fail” leads to simpler code

#103

Post by anupamchugh »

kbyatnal wrote: post #28

the best TLDR I've seen of this philosophy is - if you find yourself writing all over your codebase try { ... } catch (e) { console.log(e) } then you should probably just "let it fail" since you can't actually handle the error

That's impressive!

Re: How “let it fail” leads to simpler code

#104

Post by makeitdouble »

tsimionescu wrote: post #82
angarg12 wrote: post #64

One of the pieces of software I'm most proud of is a service to manage the dynamic part of our infrastructure. It uses control theory and let it fail to great effect. The service reads the state of the system, and applies change to converge to a configured policy. If it encounter an error, it doesn't try to handle or fix it, it just fails and logs a fine grained metric, plus a general error metric. The system fails a…

This is exactly why I think all the discussions about the importance of error handling paths (and the aversion drive have to exceptions) are usually overblown. The most successful, and common, error handling strategy is to log and abandon the whole operation, cleaning up everything the operation left around. If you have one process per operation, this is often very well captured by doing exit() at the place of the er…

That needs a few conditions to be accepted:

- an isolated process which failure doesn’t cascade other part’s failure

- as parent mentionned, where and what failed needs to be super clear

- people are available to timely react to the error, so a rerun will succeed

Failing any of the above, and you’ll need extensive and probably complex error handling that can at least help the system work in a degraded state until the error source is handled.

In my experience, it’s really the last part that motivates engineers to try to deal with the maximum error cases automatically instead of having to deal with them weekends after weekends.

Re: How “let it fail” leads to simpler code

#105

Post by secondcoming »

angarg12 wrote: post #64

One of the pieces of software I'm most proud of is a service to manage the dynamic part of our infrastructure. It uses control theory and let it fail to great effect. The service reads the state of the system, and applies change to converge to a configured policy. If it encounter an error, it doesn't try to handle or fix it, it just fails and logs a fine grained metric, plus a general error metric. The system fails a…

> fails and logs

What if your logging code was written with the same philosophy?

Re: How “let it fail” leads to simpler code

#106

Post by t0mek »

angarg12 wrote: post #97
arunaugustine wrote: post #80

Earlier quoted context omitted.

Yes, please give us more info about using control theory and how one might think about building such a system please..

These kind of systems are not always appropriate, but when they do, they work wonderfully. Our use case was to build a service to manage the dynamic part of our infrastructure. These are infra pieces that are created/deleted/modified on the fly according to some policies, instead of defining them statically as code. The implementation is simply a lambda function that runs every minute, loads a policy, compares the cu…

This approach is very similar to the way how Kubernetes custom resource reconciliation works (and Kubernetes in general, but the custom resources is the way how you can bring your own logic there).

In Kubernetes you can define your own types, Custom Resources (basically JSONs with schema) and deploy "operators" - services that should handle these new types. Every time you create or modify your custom resource, the operator is triggered and it should "reconcile" your resource.

Now this reconciliation process is stateless. It doesn't know what exactly changed in your resource, so it should just go through the list of all the things that it needs to do (create or remove pods, services, configmaps, etc.) and if something is not right (e.g. a missing service), try to make it right or fail. In any case, the output should be written in the custom resource's .status section.

There's no active waiting - if the operator sees that some other resource is not ready yet (a required pod is still starting), it should just mark your resource as not ready and finish. If the pod state changes, the next reconciliation will notice it. It should do as much as it can to bring reality to the expectation, but not more.

If implemented correctly, this is surprisingly resilient. The idempotent nature of the reconciliation loop makes it perfect for errors handling. For instance, your reconciliation may fail because some pod is not running correctly. It's nothing that your operator can fix. But if the pod auto-heals (maybe the network connectivity was restored or an external service is available again), the operator will auto-heal as well, without a manual intervention. The next reconciliation loop will just see the pod is available again and carry on.

Re: How “let it fail” leads to simpler code

#107

Post by angarg12 »

secondcoming wrote: post #105
angarg12 wrote: post #64

One of the pieces of software I'm most proud of is a service to manage the dynamic part of our infrastructure. It uses control theory and let it fail to great effect. The service reads the state of the system, and applies change to converge to a configured policy. If it encounter an error, it doesn't try to handle or fix it, it just fails and logs a fine grained metric, plus a general error metric. The system fails a…

> fails and logs What if your logging code was written with the same philosophy?

In another comment I mention that our service has some properties which make this a great solution. It doesn't always work, but when it does, it's awesome.

To your point, Erlang has the concept of supervisor trees and handling of errors. Similarly, our supervisor is the Lambda runtime. If everything goes wrong, the runtime will emit a metric error.

But what if Lambda itself fails? Or Cloudwatch? or any other supporting service? of course this is possible. But probably at that point the system is so fucked up that the fate of our little service is of little relevance. At least we know from past outages that when normal operations resume, the system will correct itself.

Re: How “let it fail” leads to simpler code

#108

Post by yawaramin »

jillesvangurp wrote: post #100
Mr_P wrote: post #14

For all the hate that Java tends to get, the language natively supports this distinction between: * Expected errors - Checked Exceptions * Unexpected errors - Unchecked Exceptions Idiomatic Java also makes heavy use of asserts, e.g. using the Guava Preconditions library.

Modern Java, should not produce a lot of checked exceptions. Unfortunately, a large part of the standard library is 25 years old and still full of things that throw checked exceptions. If you use something like Spring or Quarkus, you'll not find a lot of those. Kotlin improved on Java by treating all exceptions as unchecked. Including those from Java code. This was intentional and based on the observation that checke…

It's not clear to me that checked exceptions are actually a mistake, rather than just developers getting annoyed at their compiler forcing them to handle errors.

Just fyi, Scala predates Kotlin in not enforcing checked exceptions.

Re: How “let it fail” leads to simpler code

#109

Post by Karellen »

wsc981 wrote: post #93
arunaugustine wrote: post #80

Earlier quoted context omitted.

Yes, please give us more info about using control theory and how one might think about building such a system please..

My guess would be add assertions everywhere instead of throwing exceptions.

Why not throw exceptions, and just never use try/catch? That way, all exceptions are uncaught and should terminate the program, in a way that takes advantage of the programming language's native error reporting facilities.

Re: How “let it fail” leads to simpler code

#110

Post by jeshin »

ramraj07 wrote: post #33

A corollary or generalized interpretation of this approach (and someone please specify if there’s a formal term for this) is: “fail locally, and immediately.” What I mean is that once something unexpected happens your code should ideally fail in that step itself. The simplest most common example I’ve seen with python programmers is when they pass around dicts as arguments in complex code bases. Methods expect various…

dicts are just a little too easy to use. You just smear it down, pass it around, and you're in business. If you really want to shoot yourself in the foot, also modify its structure here and there along the way, it's just so convenient. Who needs all that hassle of declaring a data class for each little thing?

It took me a little too long to realize that a data class represents a contract about the structure of your data, meaning that no matter how many calls deep you are passing it around, you will always know its structure without having to trace it back to the origin, and that's a powerful thing.

Post reply on HN