Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't “let it crash”, let it heal

zachdaniel.dev

Post reply on HN
Print view View original

31–40 of 92 posts

Re: Don't “let it crash”, let it heal

#31

Post by davidclark »

zwnow wrote: post #20
praptak wrote: post #19

A condition that "should not happen" might still be a problem specific to a particular request. If you "just crash" it turns this request from one that only triggers a http 500 response to one that crashes the process. This increases the risk of Query of Death scenarios where the frontend that needs to serve this particular request starts retrying it with different backends and triggers restarts faster than the proce…

This is funny given Elixir/Erlangs whole idea is "let it crash". In Go I just have a Recovery Middleware for any type of problem. Don't know how other langs do it tho

I don’t know Go, but that sounds like someone has simply written part of Erlang in Go.

Re: Don't “let it crash”, let it heal

#32

Post by davidclark »

praptak wrote: post #19

A condition that "should not happen" might still be a problem specific to a particular request. If you "just crash" it turns this request from one that only triggers a http 500 response to one that crashes the process. This increases the risk of Query of Death scenarios where the frontend that needs to serve this particular request starts retrying it with different backends and triggers restarts faster than the proce…

You should try to do some load testing of a real Erlang system and compare how it handles this scenario against other languages/frameworks. What you are describing is one of the exact things the Erlang system is strong against due to the scheduler.

Re: Don't “let it crash”, let it heal

#33

Post by dns_snek »

lawn wrote: post #11
HexDecOctBin wrote: post #9

How does restarting the process fix the crash? If the process crashed because a file was missing, it will still be missing when the process is restarted. Is an infinite crash-loop considered success in Erlang?

Typically you then let the error bubble up in the supervisor tree if restarting multiple times doesn't fix it. Of course there are still errors that can't be recovered from, in which case the whole program may finally crash.

> in which case the whole program may finally crash.

This may happen if you let it, but it's basically never the desired outcome. If you were handling a user request, it should stop by returning a HTTP 500 to the client, or if you were processing a background job of some sort, it should stop with a watchdog process marking the job as a failure, not with the entire system crashing.

Re: Don't “let it crash”, let it heal

#34

Post by masklinn »

HexDecOctBin wrote: post #9

How does restarting the process fix the crash? If the process crashed because a file was missing, it will still be missing when the process is restarted. Is an infinite crash-loop considered success in Erlang?

> Is an infinite crash-loop considered success in Erlang?

Of course not, but usually that's not what happens, instead a process crashes because some condition was not considered, the corresponding request is aborted, and a supervisor restarts the process (or doesn't because the acceptor spawns a process per request / client).

Or a long-running worker got into an incorrect state and crashed, and a supervisor will restart it in a known good state (that's a pretty common thing to do in hardware, BEAM makes that idiomatic in software).

Re: Don't “let it crash”, let it heal

#36

Post by vrnvu »

goosejuice wrote: post #5

https://erlang.org/pipermail/erlang-questions/2003-March/007... The origin, as far as I know it. I think it still holds, is insightful, as a general case. Let it heal seems pretty close to what Joe was getting at.

>>This organization corresponds nicely to a idealized human organization of bosses and workers - bosses say what is to be done, workers do stuff. Bosses do quality control and check that things get done, if not they fire people re-organize and tell other people to do the stuff. If they fail (the bosses) they get sacked etc. >

We miss you Joe :)

Re: Don't “let it crash”, let it heal

#37

Post by victorbjorklund »

HexDecOctBin wrote: post #9

How does restarting the process fix the crash? If the process crashed because a file was missing, it will still be missing when the process is restarted. Is an infinite crash-loop considered success in Erlang?

Elixir dev: It does not solve all issues. But sometimes you have some kind of rare bug that just happens once X,Z and Y happens in a specific order. If it is restarted it might not happen that way again. Or it might be a temporary problem. You are reaching for an API and it temporarily has issues. It might not have it anymore in 50 ms.

But of course if it crashes because you are reading a file that does not exist it doesnt solve the issue (but it avoids crashing the whole system).

Re: Don't “let it crash”, let it heal

#38

Post by victorbjorklund »

victorbjorklund wrote: post #37
HexDecOctBin wrote: post #9

How does restarting the process fix the crash? If the process crashed because a file was missing, it will still be missing when the process is restarted. Is an infinite crash-loop considered success in Erlang?

Elixir dev: It does not solve all issues. But sometimes you have some kind of rare bug that just happens once X,Z and Y happens in a specific order. If it is restarted it might not happen that way again. Or it might be a temporary problem. You are reaching for an API and it temporarily has issues. It might not have it anymore in 50 ms. But of course if it crashes because you are reading a file that does not exist it…

Note that let is crash doesnt mean we shouldnt fix bugs. It is more about if there is a bug we havent fixed it is better to make the crash just crash a tiny part of the program than the whole program

Re: Don't “let it crash”, let it heal

#39

Post by valenterry »

refactor_master wrote: post #6

Question as a complete outsider: If I run idempotent Python applications in Kubernetes containers and they crash, Kubernetes will eventually restart them. Of course, knowing what to do on IO errors is nicer than destroying and restarting everything with a really bigger hammer (as the article also mentions, you can serve a better error message for whoever has to “deal” with the problem), but eventually they should end…

In general, if you can move any kind of logic to a lower level, that's better.

For example, testing that kubernetes restarts work correctly is tricky and requires a complicated setup. Testing that an erlang process/actor behaves as expected is basically a unit test.

Re: Don't “let it crash”, let it heal

#40

Post by creatonez »

valenterry wrote: post #14

There are a few stages, and each improves on the previous ones: 1. Detect crashes at runtime and by default stop/crash to prevent continuing with invalid program state 2. Detect crashes at runtime and handle them according to the business context (e.g. crash or retry or fallback-to or ...) to prevent bad UX through crashes. 3. Detect potential crashes at compile-time to prevent the dev from forgetting to handle them…

Based on your list there is an opportunity to define stage -1 of error handling sanity, the Eval-Rinse-Reload loop, as implemented by FuckItJS, the original Javascript Error Steamroller: https://github.com/mattdiamond/fuckitjs

> Through a process known as Eval-Rinse-Reload-And-Repeat, FuckItJS repeatedly compiles your code, detecting errors and slicing those lines out of the script. To survive such a violent process, FuckItJS reloads itself after each iteration, allowing the onerror handler to catch every single error in your terribly written code.

> [...]

> This will keep evaluating your code until all errors have been sliced off like mold on a piece of perfectly good bread. Whether or not the remaining code is even worth executing, we don't know. We also don't particularly care.

Post reply on HN