Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't “let it crash”, let it heal

zachdaniel.dev

Post reply on HN
Print view View original

71–80 of 92 posts

Re: Don't “let it crash”, let it heal

#71

Post by tmcb »

tmcb wrote: post #69
borromakot wrote: post #68

Earlier quoted context omitted.

Then I don't think you understand how the phrase is used in Elixir/Erlang. The phrase is about letting processes crash.

No need for the snarky comment. If I am wrong, that is fine. Of course Joe Armstrong could explain what I meant, but in a much better way: https://erlang.org/pipermail/erlang-questions/2003-March/007... (edit: see the "Why was error handling designed like this?" part for reference) My personal interpretation is that systems must be able to handle crashing processes gracefully. There is no benefit in letting processes…

Actually, now I thought about it, I know exactly what irked me about the approach. I hope the author takes it as constructive feedback:

Saying "let it crash is a tagline that actually means something else because the BEAM is supposed to be used in this particular way" sounds slightly "cargo-cultish", to the point where we have to challenge the meaning of the actual word to make sense of it.

Joe Armstrong's e-mail, on the other hand, says (and I paraphrase): "the BEAM was designed from the ground up to help developers avoid the creation of ad-hoc protocols for process communication, and the OTP takes that into consideration already. Make sure your system, not your process, is resilient, and literally let processes crash." Boom. There is no gotcha there. Also, there is the added benefit that developers for other platforms now understand that the rationale is justified by the way BEAM/OTP were designed and may not be applicable to their own platforms.

Re: Don't “let it crash”, let it heal

#72

Post by juped »

There's really not more that's useful to say than the relevant section (4.4) of Joe Armstrong's thesis says:

>How does our philosophy of handling errors fit in with coding practices? What kind of code must the programmer write when they find an error? The philosophy is let some other process fix the error, but what does this mean for their code? The answer is let it crash. By this I mean that in the event of an error, then the program should just crash. But what is an error? For programming purpose we can say that:

>• exceptions occur when the run-time system does not know what to do.

>• errors occur when the programmer doesn’t know what to do.

>If an exception is generated by the run-time system, but the programmer had foreseen this and knows what to do to correct the condition that caused the exception, then this is not an error. For example, opening a file which does not exist might cause an exception, but the programmer might decide that this is not an error. They therefore write code which traps this exception and takes the necessary corrective action.

>Errors occur when the programmer does not know what to do. Programmers are supposed to follow specifications, but often the specification does not say what to do and therefore the programmer does not know what to do.

>[...]

>The defensive code detracts from the pure case and confuses the reader—the diagnostic is often no better than the diagnostic which the compiler supplies automatically.

Note that this "program" is a process. For a process doing work, encountering something it can't handle is an error per the above definitions, and the process should just die, since there's nothing better for it to do; for a supervisor process supervising such processes-doing-work, "my child process exited" is an exception at worst, and usually not even an exception since the standard library supervisor code already handles that.

Re: Don't “let it crash”, let it heal

#73

Post by jonhohle »

stcg wrote: post #65

"Let it crash" is a sentence that gets attention. It makes a person want to know more about it, as it sounds controversial and different. "Let it heal" doesn't have that.

It also has a deeper philosophical meaning of unexpected software bugs should be noisy and obvious instead of causing silently corruption or misleading user experience. If monitoring doesn’t catch the failure, customers will and it can be fixed right away (whether it’s the software, a hardware error, dependency issue, etc.).

A web service returning a 500 error code is a lot more obvious than a 200 with an invalid payload. A crashed app with a stack trace is easier to debug and will cause more user feedback than an app than hangs in a retry loop.

When I had to deal with these things in the Java world, it meant not blindly handling or swallowing exceptions that business code had no business caring about. Does your account management code really think it knows how to properly handle an InterruptedException? Unless your answer is rollback and reset the interrupted flag it’s probably wrong. Can’t write a test for a particular failure scenario? That better blow up loudly with enough context that makes it possible to understand the error condition (and then write a test for it).

Re: Don't “let it crash”, let it heal

#74

Post by jlouis »

HexDecOctBin wrote: post #9

How does restarting the process fix the crash? If the process crashed because a file was missing, it will still be missing when the process is restarted. Is an infinite crash-loop considered success in Erlang?

It's not going to be missing the next time around. Usually the file is missing due to some concurrency-problem where the file only gets to exist a little later. A process restart certainly fixes this.

If the problem persists, a larger part of the supervision tree is restarted. This eventually leads to a crash of the full application, if nothing can proceed without this application existing in the Erlang release.

The key point is that there's a very large class of errors which is due to the concurrent interaction of different parts of the system. These problems often go away on the next try, because the risk of them occurring is low.

Re: Don't “let it crash”, let it heal

#75

Post by borromakot »

tmcb wrote: post #71
tmcb wrote: post #69

Earlier quoted context omitted.

No need for the snarky comment. If I am wrong, that is fine. Of course Joe Armstrong could explain what I meant, but in a much better way: https://erlang.org/pipermail/erlang-questions/2003-March/007... (edit: see the "Why was error handling designed like this?" part for reference) My personal interpretation is that systems must be able to handle crashing processes gracefully. There is no benefit in letting processes…

Actually, now I thought about it, I know exactly what irked me about the approach. I hope the author takes it as constructive feedback: Saying "let it crash is a tagline that actually means something else because the BEAM is supposed to be used in this particular way" sounds slightly "cargo-cultish", to the point where we have to challenge the meaning of the actual word to make sense of it. Joe Armstrong's e-mail, on…

If I sounded snarky that wasn't my intention. At the end of the day though it doesn't feel like you read the article which was clearly in a different context than the one in which you responded. FWIW I didn't expect this small article speaking to a small audience (Elixir devs) to make the rounds on hacker news.

I agree on the importance of defining terms, and I think the important thing here is that "process" in Joe's parlance is not an OS level process, it is one of a fleet of processes running inside the BEAM VM. And the "system" in this case is the supervisory system around it, which itself consists of individual processes.

I'm critiquing a common misunderstanding of the phrase "Let it crash", whereby effectively no local error handling is performed. This leads to worse user experiences and worse outcomes in general. I understand that you're offering critique, but it again sounds like you're critiquing a reductive element (the headline itself).

Re: Don't “let it crash”, let it heal

#76

Post by nine_k »

0x445442 wrote: post #70
nine_k wrote: post #30

Earlier quoted context omitted.

Let it crash, so that if something goes wrong, it does not do so silently. Let it crash, because a relevant manager will detect it, report it, clean it up, and restart it, without you having to write a line of code for that. Let it crash as soon as possible, so that any problem (like a crash loop) is readily visible. It's very easy to replace arbitrary bits of Erlang code in a running system, without affecting the re…

Are individual agents deployable on their own or does the entire "app" of agents need to be deployed as a single group? If individually deployable, what does this look like from a version control and a CI/CD perspective?

To the best of my knowledge: yes, individual parts are deployable separately, within reason. No, there explicitly no need to deploy the whole thing at once, and especially to shut it down all at once.

Erlang works by message passing and duck typing, so, as long as your interfaces are compatible (backwards or forwards), you can alter the implementation, and evolve the interfaces. Think microservices, but when every function can be a microservice, at an absolutely trivial cost.

Re: Don't “let it crash”, let it heal

#77

Post by tmcb »

borromakot wrote: post #75
tmcb wrote: post #71

Earlier quoted context omitted.

Actually, now I thought about it, I know exactly what irked me about the approach. I hope the author takes it as constructive feedback: Saying "let it crash is a tagline that actually means something else because the BEAM is supposed to be used in this particular way" sounds slightly "cargo-cultish", to the point where we have to challenge the meaning of the actual word to make sense of it. Joe Armstrong's e-mail, on…

If I sounded snarky that wasn't my intention. At the end of the day though it doesn't feel like you read the article which was clearly in a different context than the one in which you responded. FWIW I didn't expect this small article speaking to a small audience (Elixir devs) to make the rounds on hacker news. I agree on the importance of defining terms, and I think the important thing here is that "process" in Joe'…

I did read the article. I concede that I might not have understood it. Again, I never said it is wrong, but rather that it has a blind spot. I am familiar with Joe Armstrong’s work because I worked on a proprietary (and rather worse tbf) native distributed systems middleware in the past.

Re: Don't “let it crash”, let it heal

#78

Post by bccdee »

ramchip wrote: post #12
HexDecOctBin wrote: post #9

How does restarting the process fix the crash? If the process crashed because a file was missing, it will still be missing when the process is restarted. Is an infinite crash-loop considered success in Erlang?

I recommend https://ferd.ca/the-zen-of-erlang.html starting from "if my configuration file is corrupted, restarting won't fix anything". The tl;dr is it helps with transient bugs.

> if you feel that your well-understood regular failure case is viable, then all your error handling can fall-through to that case.

This is my favourite line, because it generalizes the underlying principle beyond the specific BEAM/OTP model in a way that carries over well to the more common sort of database-backed services that people tend to write.

Re: Don't “let it crash”, let it heal

#79

Post by bccdee »

IshKebab wrote: post #48

Ah this makes sense. I always thought "let it crash" made it sound like Elixir devs just don't bother with error checking, like writing Java without any `catch`es, or writing Rust that only uses `.unwrap()`. If they just mean "processes should be restartable" then that sounds way more reasonable. Similar idea to this but less fancy: https://flawless.dev/ It's a pretty terrible slogan if it makes your language sound w…

I've been seeing a lot of these durable workflow engines around lately, for some reason. I'm not sure I understand the pitch. It just seems like a thin wrapper around some very normal patterns for running background jobs. Persist your jobs in a db, checkpoint as necessary, periodically retry. I guess they're meant to be a low-code alternative to writing the db tables yourself, but it seems like you're not saving much code in practice.

Re: Don't “let it crash”, let it heal

#80

Post by jfengel »

Quekid5 wrote: post #51
corysama wrote: post #16

Earlier quoted context omitted.

I’m only an armchair expert on Erlang. But, having looked into it repeatedly for a couple decades, my take-away is the “Let it crash” slogan is good. But, also presented a bit out of context. Or, at least assuming context that most people don’t have. Erlang is used in situations involving a zillion incoming requests. If an individual request fails… Maybe it was important. Maybe it wasn’t. If it was important, it’s ex…

> You can pull this off in other languages via careful attention to the details of your request-handling code. But, the creators of the Erlang language and foundational frameworks have set their users up for success via careful attention to the design of the system as a whole. +10. So many people miss this very important point. If you have lots of mutable shared state, or can accidentally leak such into your actor co…

The alternative to straight-line code used to be called "spaghetti code".

There was a joke article parodying "GOTO considered harmful" by suggesting a "COME FROM" command. But in a lot of always, that's exactly what many modern frameworks and languages aim for.

Post reply on HN