Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How “let it fail” leads to simpler code

yiming.dev

Post reply on HN
Print view View original

111–120 of 165 posts

Re: How “let it fail” leads to simpler code

#111

Post by gabeio »

kgeist wrote: post #99

>ignore unexpected exceptions Isn't it the way it already is in practice, not something specific to Erlang? If an exception is unexpected, usually there won't be an exception handler for it, otherwise a developer pretty much expected it. Developers are generally lazy so in my practice the default is usually to let it fail, and there's usually going to be an exception handler that does something other than logging and…

> If an exception is unexpected, usually there won't be an exception handler for it

That honestly depends on how the language and program are written, python is a great and horrible example of where you can handle any exception even ones that were just created by the program:

    try:
        crash_hard_here()
    except: # by default (and unfortunately) will catch *everything*
        pass # and this is one of the worst offenders inside an except, to just outright ignore the exception and continue as if nothing happened and to not even log it.
I can not tell you the amount of production code where I've seen catch all exceptions, and they are the lazy way to know something will not "crash" even though much worse things can happen now.

Re: How “let it fail” leads to simpler code

#112

Post by tsimionescu »

makeitdouble wrote: post #104
tsimionescu wrote: post #82

Earlier quoted context omitted.

This is exactly why I think all the discussions about the importance of error handling paths (and the aversion drive have to exceptions) are usually overblown. The most successful, and common, error handling strategy is to log and abandon the whole operation, cleaning up everything the operation left around. If you have one process per operation, this is often very well captured by doing exit() at the place of the er…

That needs a few conditions to be accepted: - an isolated process which failure doesn’t cascade other part’s failure - as parent mentionned, where and what failed needs to be super clear - people are available to timely react to the error, so a rerun will succeed Failing any of the above, and you’ll need extensive and probably complex error handling that can at least help the system work in a degraded state until the…

I believe one of the assumptions is that the failed process can automatically restart (e.g. using systemd, Kubernetes, hypervisor policies, top-level retries) - so that transient errors recover automatically, and at worst cost some performance or tiny bits of lost work (e.g. the setting an end user just hit apply on doesn't get applied, so they have to click again).

Re: How “let it fail” leads to simpler code

#113

Post by Too »

arunaugustine wrote: post #80
angarg12 wrote: post #64

One of the pieces of software I'm most proud of is a service to manage the dynamic part of our infrastructure. It uses control theory and let it fail to great effect. The service reads the state of the system, and applies change to converge to a configured policy. If it encounter an error, it doesn't try to handle or fix it, it just fails and logs a fine grained metric, plus a general error metric. The system fails a…

Yes, please give us more info about using control theory and how one might think about building such a system please..

This is how Kubernetes works in many ways. Crash a pod and the control loop inside the ReplicaSet will create a new pod for you. Scaling nodes is based on similar principles of desired vs actual values.

Re: How “let it fail” leads to simpler code

#114

Post by e9 »

at_a_remove wrote: post #89

I struggle to find the correct descriptor for a counter-example, wherein You Really Want Success for the process as a whole, but it is acceptable for a sliver of it to fail, in the the context of ETL. I have an ETL I am told (I switched jobs) that is still working, from 2008. It was built to be a tank, and I also did another forbidden thing: Pokemon Exception Handling. It's a guideline, not a law of physics, and it i…

Funny thing, I had same experience. I also built robust ETL. It was for ingesting and manipulating financial data from 30+ different banks and I also did Pokemon Exception Handling to make it robust.

In general my philosophy is: I don't want to wake up at 4am unless it's urgent. What can I do to gracefully handle failures to achieve that goal?

Re: How “let it fail” leads to simpler code

#116

Post by Too »

bzxcvbn wrote: post #57
aunderscored wrote: post #40

Earlier quoted context omitted.

Python is strongly typed. You want statically typed. (Instead of duck typed / dynamically typed)

Can you guess what this code does? class foo: pass obj = foo() obj.bar = "I thought Python was strongly typed?" print(obj.bar) And even better: class foo: a = 42 obj = foo() print(obj.a) del foo.a print(obj.a) Whatever your opinion on what the imprecise sentence "strongly typed language" should mean, these are definitely not features of one.

No need to guess, IDE is flashing bright red and mypy screaming main.py:5: error: "foo" has no attribute "bar". One could still say fuckit and run it anyway, but why would you take that risk. This would never get through to production.

Re: How “let it fail” leads to simpler code

#117

Post by kortilla »

angarg12 wrote: post #64

One of the pieces of software I'm most proud of is a service to manage the dynamic part of our infrastructure. It uses control theory and let it fail to great effect. The service reads the state of the system, and applies change to converge to a configured policy. If it encounter an error, it doesn't try to handle or fix it, it just fails and logs a fine grained metric, plus a general error metric. The system fails a…

This is just kubernetes right? Declarative desired state model. Containers created and destroyed to get there. Crashes happen, metrics are incremented, load balancers route around the crashing pod until they recover (or are replaced), etc.

Re: How “let it fail” leads to simpler code

#118

Post by IshKebab »

TameAntelope wrote: post #23
smackeyacky wrote: post #12

I don't agree with this approach. Say you have a network service that relies on other network services. It is not difficult to write those such that they know to back off / retry when something disappears. It's extremely useful in a lot of situations: if you do work on a laptop that gets regularly unplugged, having running test services that know to reconnect makes your life easier. In production, having things autom…

What you're describing are "known" states; the idea behind "let it fail" is that you shouldn't write code that exhaustively handles every single potential outcome, just the ones that are part of your code's path in general use. Definitely write code to handle network issues. Don't write code to handle random bitflips, ways to handle garbage coming back from the service you're connecting to, or try to handle OOM error…

> Don't write code to handle random bitflips

It depends what you're doing. There's no fixed threshold for "errors that you should handle" so smackeyacky is right - handle the errors you can (but don't spend an inordinate amount of time handling very unlikely errors).

Bitflips are not very unlikely on huge systems so you need to handle them.

In my experience trying to distinguish between "expected" or "normal" errors and "unexpected" or "exceptional" errors is pointless and impossible. It's better to think about the likelihood of errors.

Re: How “let it fail” leads to simpler code

#119

Post by blub »

kgeist wrote: post #99

>ignore unexpected exceptions Isn't it the way it already is in practice, not something specific to Erlang? If an exception is unexpected, usually there won't be an exception handler for it, otherwise a developer pretty much expected it. Developers are generally lazy so in my practice the default is usually to let it fail, and there's usually going to be an exception handler that does something other than logging and…

In C++ for example there’s an exception hierarchy and several classes of errors similar to what you enumerated, so the handler might catch a specific exception as a more generic one. There’s also … which catches all exceptions.

This is fine, as one has the choice of exposing error details to outside code or not.

Re: How “let it fail” leads to simpler code

#120

Post by blub »

gabeio wrote: post #111
kgeist wrote: post #99

>ignore unexpected exceptions Isn't it the way it already is in practice, not something specific to Erlang? If an exception is unexpected, usually there won't be an exception handler for it, otherwise a developer pretty much expected it. Developers are generally lazy so in my practice the default is usually to let it fail, and there's usually going to be an exception handler that does something other than logging and…

> If an exception is unexpected, usually there won't be an exception handler for it That honestly depends on how the language and program are written, python is a great and horrible example of where you can handle any exception even ones that were just created by the program: try: crash_hard_here() except: # by default (and unfortunately) will catch *everything* pass # and this is one of the worst offenders inside an…

This catch and ignore seems to be straight out of the (sadly not real) Visual Basic design patterns book :-)
Post reply on HN