Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't “let it crash”, let it heal

zachdaniel.dev

Post reply on HN
Print view View original

41–50 of 92 posts

Re: Don't “let it crash”, let it heal

#41

Post by valenterry »

creatonez wrote: post #40
valenterry wrote: post #14

There are a few stages, and each improves on the previous ones: 1. Detect crashes at runtime and by default stop/crash to prevent continuing with invalid program state 2. Detect crashes at runtime and handle them according to the business context (e.g. crash or retry or fallback-to or ...) to prevent bad UX through crashes. 3. Detect potential crashes at compile-time to prevent the dev from forgetting to handle them…

Based on your list there is an opportunity to define stage -1 of error handling sanity, the Eval-Rinse-Reload loop, as implemented by FuckItJS, the original Javascript Error Steamroller: https://github.com/mattdiamond/fuckitjs > Through a process known as Eval-Rinse-Reload-And-Repeat , FuckItJS repeatedly compiles your code, detecting errors and slicing those lines out of the script. To survive such a violent process…

Oh, thank you for the nostalgic reminder of that one. I read that a decade ago and found it hilarious.

Re: Don't “let it crash”, let it heal

#42

Post by hesus_ruiz »

It is very strange that a post trying to explain the concept of "let it crash" in Elixir (which runs on the BEAM VM) does not mention the doctoral thesis of Joe Armstrong: "Making reliable distributed systems in the presence of software errors".

It must be compulsory lecture for anybody interested in reliable systems, even if they do not use the BEAM VM.

https://www.diva-portal.org/smash/record.jsf?pid=diva2%3A104...

Re: Don't “let it crash”, let it heal

#43

Post by conradfr »

HexDecOctBin wrote: post #9

How does restarting the process fix the crash? If the process crashed because a file was missing, it will still be missing when the process is restarted. Is an infinite crash-loop considered success in Erlang?

If the rest of the program is still running while you fix it, yes?

Also, restarting endlessly is just one strategy between multiple others.

Re: Don't “let it crash”, let it heal

#45

Post by gopher_space »

masklinn wrote: post #34
HexDecOctBin wrote: post #9

How does restarting the process fix the crash? If the process crashed because a file was missing, it will still be missing when the process is restarted. Is an infinite crash-loop considered success in Erlang?

> Is an infinite crash-loop considered success in Erlang? Of course not, but usually that's not what happens, instead a process crashes because some condition was not considered, the corresponding request is aborted, and a supervisor restarts the process (or doesn't because the acceptor spawns a process per request / client). Or a long-running worker got into an incorrect state and crashed, and a supervisor will rest…

Both of your examples look like infinite crash-loops if your work needs to be correct more than it needs to be available. E.g. there aren't any known good states prior to an unexpected crash, you're just throwing a hail mary because the alternatives are impractical.

Re: Don't “let it crash”, let it heal

#46

Post by emoII »

valenterry wrote: post #39
refactor_master wrote: post #6

Question as a complete outsider: If I run idempotent Python applications in Kubernetes containers and they crash, Kubernetes will eventually restart them. Of course, knowing what to do on IO errors is nicer than destroying and restarting everything with a really bigger hammer (as the article also mentions, you can serve a better error message for whoever has to “deal” with the problem), but eventually they should end…

In general, if you can move any kind of logic to a lower level, that's better. For example, testing that kubernetes restarts work correctly is tricky and requires a complicated setup. Testing that an erlang process/actor behaves as expected is basically a unit test.

I bet the kubernetes project has test for that, why should I as an application developer care about testing something other than my own code?

Re: Don't “let it crash”, let it heal

#47

Post by valenterry »

emoII wrote: post #46
valenterry wrote: post #39

Earlier quoted context omitted.

In general, if you can move any kind of logic to a lower level, that's better. For example, testing that kubernetes restarts work correctly is tricky and requires a complicated setup. Testing that an erlang process/actor behaves as expected is basically a unit test.

I bet the kubernetes project has test for that, why should I as an application developer care about testing something other than my own code?

Oh of course, I'm sure the kubernetes project tests that they trigger restarts correctly etc.

But that doesn't cover the behavior of your app, the specific configuration you ask kubernetes to use and how the app uses its health endpoints etc. - this is all purely about your own code/config, the kubernetes team can't test that.

Re: Don't “let it crash”, let it heal

#48

Post by IshKebab »

Ah this makes sense. I always thought "let it crash" made it sound like Elixir devs just don't bother with error checking, like writing Java without any `catch`es, or writing Rust that only uses `.unwrap()`.

If they just mean "processes should be restartable" then that sounds way more reasonable. Similar idea to this but less fancy: https://flawless.dev/

It's a pretty terrible slogan if it makes your language sound worse than it actually is.

Re: Don't “let it crash”, let it heal

#49

Post by tmcb »

It is very common to interpret taglines by their face value, and I believe the author did just that, although the point brought up is valid.

In order to “let it crash”, we must design the system in a way that crashes would not be catastrophic, stability wise. Letting it crash is not a commandment, though: it is a reminder that, in most cases, a smart healing strategy might be overkill.

Re: Don't “let it crash”, let it heal

#50

Post by borromakot »

tmcb wrote: post #49

It is very common to interpret taglines by their face value, and I believe the author did just that, although the point brought up is valid. In order to “let it crash”, we must design the system in a way that crashes would not be catastrophic, stability wise. Letting it crash is not a commandment, though: it is a reminder that, in most cases, a smart healing strategy might be overkill.

Author: I'm literally explaining not to interpret the tag line at face value.
Post reply on HN