Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't “let it crash”, let it heal

zachdaniel.dev

Post reply on HN
Print view View original

51–60 of 92 posts

Re: Don't “let it crash”, let it heal

#51

Post by Quekid5 »

corysama wrote: post #16
HexDecOctBin wrote: post #9

How does restarting the process fix the crash? If the process crashed because a file was missing, it will still be missing when the process is restarted. Is an infinite crash-loop considered success in Erlang?

I’m only an armchair expert on Erlang. But, having looked into it repeatedly for a couple decades, my take-away is the “Let it crash” slogan is good. But, also presented a bit out of context. Or, at least assuming context that most people don’t have. Erlang is used in situations involving a zillion incoming requests. If an individual request fails… Maybe it was important. Maybe it wasn’t. If it was important, it’s ex…

> You can pull this off in other languages via careful attention to the details of your request-handling code. But, the creators of the Erlang language and foundational frameworks have set their users up for success via careful attention to the design of the system as a whole.

+10. So many people miss this very important point. If you have lots of mutable shared state, or can accidentally leak such into your actor code then the whole actor/supervision tree thing falls over very easily... because you can't just restart any actor without worrying about the rest of the system.

I think this is a large (but not the only[0]) part of why actors/supervisors haven't really caught on anywhere outside of Erlang, even for problem spaces where they would be suitable.

[0] I personally feel the model is very hard to reason about compared to threaded/blocking straight-line code using e.g. structured concurrency, but that may just be a me thing.

Re: Don't “let it crash”, let it heal

#52

Post by borromakot »

zwnow wrote: post #28
knome wrote: post #24

Earlier quoted context omitted.

erlang doesn't crash the program, it crashes the thread. erlang has a layered management system built in as part of OTP (open telecom platform, erlang was built for running highly concurrent telephony hardware). when a thread crashes, it dies and signals its parent. the parent then decides what to do. usually, that's just restarting the worker. maybe if ten workers have crashed in a minute, the manager itself will di…

It's not a misconception given that Elixir Forum and its Discords members will say that to you. Also I never assumed the whole program crashed so why would you explain this to me? Why would one Blog guy know it better than a lot of other Elixir devs?

Blog guy here: I do, in fact, know it better than a lot of other Elixir devs.

Re: Don't “let it crash”, let it heal

#53

Post by johnisgood »

IshKebab wrote: post #48

Ah this makes sense. I always thought "let it crash" made it sound like Elixir devs just don't bother with error checking, like writing Java without any `catch`es, or writing Rust that only uses `.unwrap()`. If they just mean "processes should be restartable" then that sounds way more reasonable. Similar idea to this but less fancy: https://flawless.dev/ It's a pretty terrible slogan if it makes your language sound w…

As someone has linked it: https://erlang.org/pipermail/erlang-questions/2003-March/007...

It is about self-healing, too.

Re: Don't “let it crash”, let it heal

#54

Post by tmcb »

borromakot wrote: post #50
tmcb wrote: post #49

It is very common to interpret taglines by their face value, and I believe the author did just that, although the point brought up is valid. In order to “let it crash”, we must design the system in a way that crashes would not be catastrophic, stability wise. Letting it crash is not a commandment, though: it is a reminder that, in most cases, a smart healing strategy might be overkill.

Author: I'm literally explaining not to interpret the tag line at face value.

Maybe I didn’t make myself clear. “Let it crash” is not something that should be thought of at the component level, it should be thought of at the system level. The fact that the application crashes “gracefully” or not is not what is really important. You should design the system in a crash-friendly way, and not to write the application and think: “oh, I believe it is OK to let it crash here”.

Re: Don't “let it crash”, let it heal

#55

Post by masklinn »

gopher_space wrote: post #45
masklinn wrote: post #34

Earlier quoted context omitted.

> Is an infinite crash-loop considered success in Erlang? Of course not, but usually that's not what happens, instead a process crashes because some condition was not considered, the corresponding request is aborted, and a supervisor restarts the process (or doesn't because the acceptor spawns a process per request / client). Or a long-running worker got into an incorrect state and crashed, and a supervisor will rest…

Both of your examples look like infinite crash-loops if your work needs to be correct more than it needs to be available. E.g. there aren't any known good states prior to an unexpected crash, you're just throwing a hail mary because the alternatives are impractical.

> there aren't any known good states prior to an unexpected crash

If there aren't any good states then the program straight up doesn't work in the first place, which gets diagnosed pretty quickly before it hits the field.

> your work needs to be correct more than it needs to be available.

"correctness over availability" tends to not be a thing, if you assume you can reach perfect and full correctness then either you never release or reality quickly proves you wrong in the field. So maximally resilient and safe systems generally plan for errors happening and how to recover from them instead of assuming they don't. There are very few fully proven non-trivial programs, and there were even less 40 years ago.

And Erlang / BEAM was designed in a telecom context, so availability is the prime directive. Which is also why distribution is built-in: if you have a single machine and it crashes you have nothing.

Re: Don't “let it crash”, let it heal

#56

Post by Muromec »

borromakot wrote: post #50
tmcb wrote: post #49

It is very common to interpret taglines by their face value, and I believe the author did just that, although the point brought up is valid. In order to “let it crash”, we must design the system in a way that crashes would not be catastrophic, stability wise. Letting it crash is not a commandment, though: it is a reminder that, in most cases, a smart healing strategy might be overkill.

Author: I'm literally explaining not to interpret the tag line at face value.

Yeah, but it's internet forum and for opinion pieces people first read comments and then maybe read the article if it's interesting.

Re: Don't “let it crash”, let it heal

#57

Post by Muromec »

gopher_space wrote: post #45
masklinn wrote: post #34

Earlier quoted context omitted.

> Is an infinite crash-loop considered success in Erlang? Of course not, but usually that's not what happens, instead a process crashes because some condition was not considered, the corresponding request is aborted, and a supervisor restarts the process (or doesn't because the acceptor spawns a process per request / client). Or a long-running worker got into an incorrect state and crashed, and a supervisor will rest…

Both of your examples look like infinite crash-loops if your work needs to be correct more than it needs to be available. E.g. there aren't any known good states prior to an unexpected crash, you're just throwing a hail mary because the alternatives are impractical.

If it has no good states you probably know it before deploying to production.

Re: Don't “let it crash”, let it heal

#58

Post by Muromec »

dns_snek wrote: post #33
lawn wrote: post #11

Earlier quoted context omitted.

Typically you then let the error bubble up in the supervisor tree if restarting multiple times doesn't fix it. Of course there are still errors that can't be recovered from, in which case the whole program may finally crash.

> in which case the whole program may finally crash. This may happen if you let it, but it's basically never the desired outcome. If you were handling a user request, it should stop by returning a HTTP 500 to the client, or if you were processing a background job of some sort, it should stop with a watchdog process marking the job as a failure, not with the entire system crashing.

returning HTTP 500 as early as possible is an example of "let it crash" approach outside of Erlang.

Re: Don't “let it crash”, let it heal

#59

Post by tmcb »

Muromec wrote: post #56
borromakot wrote: post #50

Earlier quoted context omitted.

Author: I'm literally explaining not to interpret the tag line at face value.

Yeah, but it's internet forum and for opinion pieces people first read comments and then maybe read the article if it's interesting.

I actually skimmed the article before posting. I have some exposure to Erlang, but not to Elixir. As I’ve already mentioned, I think the author’s covering of application behavior is OK, but there is more to the tagline than meets the eye.

Re: Don't “let it crash”, let it heal

#60

Post by zmgsabst »

IshKebab wrote: post #48

Ah this makes sense. I always thought "let it crash" made it sound like Elixir devs just don't bother with error checking, like writing Java without any `catch`es, or writing Rust that only uses `.unwrap()`. If they just mean "processes should be restartable" then that sounds way more reasonable. Similar idea to this but less fancy: https://flawless.dev/ It's a pretty terrible slogan if it makes your language sound w…

I think it’s more subtle:

Imagine that you’re trying to access an API, which for some reason fails.

“Let it crash” isn’t an argument against handling the timeout, but rather that you should only retry a few, bounded times rather than (eg) exponentially back off indefinitely.

When you design from that perspective, you just fail your request processing (returning the request to the queue) and make that your manager’s problem. Your managing process can then restart you, reassign the work to healthy workers, etc. If your manager can’t get things working and the queue overflows, it throws it into dead letters and crashes. That might restart the server, it might page oncall, etc.

The core idea is that within your business logic is the wrong place to handle system health — and that many problems can be solved by routing around problems (ie, give task to a healthy worker) or restarting a process. A process should crash when it isn’t scoped to handle the problem it’s facing (eg, server OOM, critical dependency offline, bad permissions). Crashing escalates the problem until somebody can resolve it.

Post reply on HN