Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't “let it crash”, let it heal

zachdaniel.dev

Post reply on HN
Print view View original

61–70 of 92 posts

Re: Don't “let it crash”, let it heal

#61

Post by vendiddy »

IshKebab wrote: post #48

Ah this makes sense. I always thought "let it crash" made it sound like Elixir devs just don't bother with error checking, like writing Java without any `catch`es, or writing Rust that only uses `.unwrap()`. If they just mean "processes should be restartable" then that sounds way more reasonable. Similar idea to this but less fancy: https://flawless.dev/ It's a pretty terrible slogan if it makes your language sound w…

I think the slogan was meant to be provocative but unfortunately it has been misinterpreted more often than not.

For example, imagine you're working with a 3rd party API and, according to the documentation, it is supposed to return responses in a certain format. What if suddenly that API stops working? Or what if the format changes?

You could write code to handle that "what if" scenario, but then trying to handle every hypothetical your code becomes bloated, more complicated, and hard to understand.

So in these cases, you accept that the system will crash. But to ensure reliability, you don't want to bring down the whole system. So there are primitives that let you control the blast radius of the crash if something unexpected happens.

Let it crash does not mean you skip validating user input. Those are issues that you expect to happen. You handle those just as you would in any programming language.

Re: Don't “let it crash”, let it heal

#62

Post by sarchertech »

zwnow wrote: post #28
knome wrote: post #24

Earlier quoted context omitted.

erlang doesn't crash the program, it crashes the thread. erlang has a layered management system built in as part of OTP (open telecom platform, erlang was built for running highly concurrent telephony hardware). when a thread crashes, it dies and signals its parent. the parent then decides what to do. usually, that's just restarting the worker. maybe if ten workers have crashed in a minute, the manager itself will di…

It's not a misconception given that Elixir Forum and its Discords members will say that to you. Also I never assumed the whole program crashed so why would you explain this to me? Why would one Blog guy know it better than a lot of other Elixir devs?

It’s well known among elixir devs that for reasons unkown, Elixir Forum is populated predominantly by people who don’t know what they’re talking about.

Re: Don't “let it crash”, let it heal

#63

Post by vendiddy »

snickerbockers wrote: post #17

>When people say “let it crash”, they are referring to the fact that practically any exited process in your application will be subsequently restarted. Because of this, you can often be much less defensive around unexpected errors. You will see far fewer try/rescue, or matching on error states in Elixir code. I just threw up in my mouth when I read this. I've never used this language so maybe my experience doesn't ap…

If get a chance to read some Elixir/Erlang code you'll see that pattern matching is used frequently to assert expected error codes. It does not mean ignore errors.

This is a common misunderstanding because unfortunately the slogan is frequently misinterpreted.

Re: Don't “let it crash”, let it heal

#64

Post by sarchertech »

praptak wrote: post #19

A condition that "should not happen" might still be a problem specific to a particular request. If you "just crash" it turns this request from one that only triggers a http 500 response to one that crashes the process. This increases the risk of Query of Death scenarios where the frontend that needs to serve this particular request starts retrying it with different backends and triggers restarts faster than the proce…

> If you "just crash" it turns this request from one that only triggers a http 500 response to one that crashes the process.

In phoenix each request has its own process and crashing that process will result in a 500 being sent to the client.

Re: Don't “let it crash”, let it heal

#66

Post by dmsnell »

gopher_space wrote: post #45
masklinn wrote: post #34

Earlier quoted context omitted.

> Is an infinite crash-loop considered success in Erlang? Of course not, but usually that's not what happens, instead a process crashes because some condition was not considered, the corresponding request is aborted, and a supervisor restarts the process (or doesn't because the acceptor spawns a process per request / client). Or a long-running worker got into an incorrect state and crashed, and a supervisor will rest…

Both of your examples look like infinite crash-loops if your work needs to be correct more than it needs to be available. E.g. there aren't any known good states prior to an unexpected crash, you're just throwing a hail mary because the alternatives are impractical.

When a process crashes, its supervisor restarts it according to some policy. These specify whether to restart the sibling process in their startup order or to only restart the crashed process.

But a supervisor also sets limits, like “10 restarts in a timespan of 1 second.” Once the limits are reached, the supervisor crashes. Supervisors have supervisors.

In this scenario the fault cascades upward through the system, triggering more broad restarts and state-reinitializations until the top-level supervisor crashes and takes the entire system down with it.

An example might bee losing a connection to the database. It’s not an expected fault to fail while querying it, so you let it crash. That kills the web request, but then the web server ends up crashing too because too many requests failed, then a task runner fails for similar reasons. The logger is still reporting all this because it’s a separate process tree, and the top-level app supervisor ends up restarting the entire thing. It shuts everything off, tries to restart the database connection, and if that works everything will continue, but if not, the system crashes completely.

Expected faults are not part of “let it crash.” E.g. if a user supplies a bad file path or network resource. The distinction is subjective and based around the expectations of the given app. Failure to read some asset included in the distribution is both unlikely and unrecoverable, so “let it crash” allows the code to be simpler in the happy path without giving up fault handling or burying errors deeper into the app or data.

Re: Don't “let it crash”, let it heal

#67

Post by tombert »

vrnvu wrote: post #36
goosejuice wrote: post #5

https://erlang.org/pipermail/erlang-questions/2003-March/007... The origin, as far as I know it. I think it still holds, is insightful, as a general case. Let it heal seems pretty close to what Joe was getting at.

>>This organization corresponds nicely to a idealized human organization of bosses and workers - bosses say what is to be done, workers do stuff. Bosses do quality control and check that things get done, if not they fire people re-organize and tell other people to do the stuff. If they fail (the bosses) they get sacked etc. > We miss you Joe :)

He was one of my favorite humans; the few emails I exchanged with him were funny and insightful.

Re: Don't “let it crash”, let it heal

#68

Post by borromakot »

tmcb wrote: post #54
borromakot wrote: post #50

Earlier quoted context omitted.

Author: I'm literally explaining not to interpret the tag line at face value.

Maybe I didn’t make myself clear. “Let it crash” is not something that should be thought of at the component level, it should be thought of at the system level. The fact that the application crashes “gracefully” or not is not what is really important. You should design the system in a crash-friendly way, and not to write the application and think: “oh, I believe it is OK to let it crash here”.

Then I don't think you understand how the phrase is used in Elixir/Erlang. The phrase is about letting processes crash.

Re: Don't “let it crash”, let it heal

#69

Post by tmcb »

borromakot wrote: post #68
tmcb wrote: post #54

Earlier quoted context omitted.

Maybe I didn’t make myself clear. “Let it crash” is not something that should be thought of at the component level, it should be thought of at the system level. The fact that the application crashes “gracefully” or not is not what is really important. You should design the system in a crash-friendly way, and not to write the application and think: “oh, I believe it is OK to let it crash here”.

Then I don't think you understand how the phrase is used in Elixir/Erlang. The phrase is about letting processes crash.

No need for the snarky comment. If I am wrong, that is fine.

Of course Joe Armstrong could explain what I meant, but in a much better way: https://erlang.org/pipermail/erlang-questions/2003-March/007... (edit: see the "Why was error handling designed like this?" part for reference)

My personal interpretation is that systems must be able to handle crashing processes gracefully. There is no benefit in letting processes crash just for the sake of it.

Re: Don't “let it crash”, let it heal

#70

Post by 0x445442 »

nine_k wrote: post #30
corysama wrote: post #16

Earlier quoted context omitted.

I’m only an armchair expert on Erlang. But, having looked into it repeatedly for a couple decades, my take-away is the “Let it crash” slogan is good. But, also presented a bit out of context. Or, at least assuming context that most people don’t have. Erlang is used in situations involving a zillion incoming requests. If an individual request fails… Maybe it was important. Maybe it wasn’t. If it was important, it’s ex…

Let it crash, so that if something goes wrong, it does not do so silently. Let it crash, because a relevant manager will detect it, report it, clean it up, and restart it, without you having to write a line of code for that. Let it crash as soon as possible, so that any problem (like a crash loop) is readily visible. It's very easy to replace arbitrary bits of Erlang code in a running system, without affecting the re…

Are individual agents deployable on their own or does the entire "app" of agents need to be deployed as a single group? If individually deployable, what does this look like from a version control and a CI/CD perspective?
Post reply on HN