Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't “let it crash”, let it heal

zachdaniel.dev

Post reply on HN
Print view View original

21–30 of 92 posts

Re: Don't “let it crash”, let it heal

#21

Post by josevalim »

snickerbockers wrote: post #17

>When people say “let it crash”, they are referring to the fact that practically any exited process in your application will be subsequently restarted. Because of this, you can often be much less defensive around unexpected errors. You will see far fewer try/rescue, or matching on error states in Elixir code. I just threw up in my mouth when I read this. I've never used this language so maybe my experience doesn't ap…

That’s actually a good example. Imagine someone forgot to check the error code from an API response. In some languages, they may attempt to parse it as if it was successful request, and succeed, leading to a result with nulls, empty arrays, or missing data that then spreads through the system. In Elixir, parsing would most likely fail thanks to pattern matching [1] and if it by any chance that fails in a core part of the system, that failure will be isolated and that particular component can be restarted.

Elixir is not about willingly ignoring error codes or failure scenarios. It is about naturally limiting the blast radius of errors without a need to program defensively (as in writing code for scenarios you don’t know “just in case”).

1: https://dashbit.co/blog/writing-assertive-code-with-elixir

Re: Don't “let it crash”, let it heal

#22

Post by cyberax »

praptak wrote: post #19

A condition that "should not happen" might still be a problem specific to a particular request. If you "just crash" it turns this request from one that only triggers a http 500 response to one that crashes the process. This increases the risk of Query of Death scenarios where the frontend that needs to serve this particular request starts retrying it with different backends and triggers restarts faster than the proce…

"Let it crash" in Erlang/Elixir means that the process that serves the request is allowed to crash. It then will be restarted by the supervisor.

Supervisors themselves form a tree, so for a crash to take down the whole app, it needs to propagate all the way to the top.

Another explanation for people familiar with exceptions in other languages: "Don't try to catch the exception inside a request handler".

Re: Don't “let it crash”, let it heal

#23

Post by josevalim »

praptak wrote: post #19

A condition that "should not happen" might still be a problem specific to a particular request. If you "just crash" it turns this request from one that only triggers a http 500 response to one that crashes the process. This increases the risk of Query of Death scenarios where the frontend that needs to serve this particular request starts retrying it with different backends and triggers restarts faster than the proce…

Processes can be marked as temporary, which means they are not restarted, and that’s what is used when managing http connections, as you can’t really restart a request on the server without the client. So the scenario above wouldn’t happen.

You still want those processes to crash though, as it allows it to automatically clean up any concurrent work. For example, if during a request you start three processes to do concurrent work, like fetching APIs, then the request process crashes, the concurrent processes are automatically cleaned up.

Re: Don't “let it crash”, let it heal

#24

Post by knome »

zwnow wrote: post #20
praptak wrote: post #19

A condition that "should not happen" might still be a problem specific to a particular request. If you "just crash" it turns this request from one that only triggers a http 500 response to one that crashes the process. This increases the risk of Query of Death scenarios where the frontend that needs to serve this particular request starts retrying it with different backends and triggers restarts faster than the proce…

This is funny given Elixir/Erlangs whole idea is "let it crash". In Go I just have a Recovery Middleware for any type of problem. Don't know how other langs do it tho

erlang doesn't crash the program, it crashes the thread. erlang has a layered management system built in as part of OTP (open telecom platform, erlang was built for running highly concurrent telephony hardware). when a thread crashes, it dies and signals its parent. the parent then decides what to do. usually, that's just restarting the worker. maybe if ten workers have crashed in a minute, the manager itself will die and restart. issues bubble up, and managers restart subsystems automatically. for some things, like parsing user data, you might never cause the manager to die, and just always restart the worker.

the article, if you should choose to read it, is explaining that people have the misconception you appear to be having due to the 'let it fail' catchphrase. it goes into detail about this system, when failing is appropriate, and when trying to work around errors is appropriate.

as erlang uses greenthreads, restarting a thread for a user API is effectively instant and free.

Re: Don't “let it crash”, let it heal

#25

Post by atoav »

The truth is that different errors have to lead to different results if you want a good organisational outcome. These could be:

- Fundamental/Fatal error: something without the process cannot function, e.g. we are missing an essential config option. Exiting with an error is totally adequate. You can't just heal from that as it would involve guessing information you don't have. Admins need to fix it

- Critical error: something that should not ever occur, e.g. having an active user without password and email. You don't exit, you skip it if thst is possible and ensure the first occurance is logged and admins are contacted

- Expected/Regular error: something that is expected to happen during the normal operations of the service, e.g. the other server you make requests to is being restarted and thus unreachable. Here the strategy may vary, but it could be something like retrying with random exponential backoff. Or you could briefly accept the values provided by that server are unknown and periodically retry to fill the unknown values. Or you could escalate that into a critical error after a certain amount of retries.

- Warnings: These are usually about something being not exactly ideal, but do not impede with the flow of the program at all. Usually has to do with bad data quality

If you can proceed without degrading the integrity of the system you should, the next thing is to decide jow important it is for humans to hear about it.

Re: Don't “let it crash”, let it heal

#26

Post by toast0 »

snickerbockers wrote: post #17

>When people say “let it crash”, they are referring to the fact that practically any exited process in your application will be subsequently restarted. Because of this, you can often be much less defensive around unexpected errors. You will see far fewer try/rescue, or matching on error states in Elixir code. I just threw up in my mouth when I read this. I've never used this language so maybe my experience doesn't ap…

Ok, so it's not really that you're not checking error codes. It's that you can write stuff like

   ok = whatever().
If whatever is successful and idomatic, it returns ok, or maybe a tuple of {ok, SomeReturn}. In that case, execution would continue. If it returns an error tuple like {error, Reason}... "Let it crash" says you can just let it crash... You didn't have anything better to do, the built in crash because {error, Reason} will do fine.

Or you could do a

   case whatever of
      ok -> ok;
      {error, nxdomain} -> ok
   end.
If it was fine to get nxdomain error, but any other error isn't acceptable... It will just crash, and that's good or at least ok. Better than having to enumerate all the possible errors, or having a catchall that then explicitly throws an eeror. It's especially hard to enumerate all possible errors because the running system can change and may return a new error that wasn't enumerated when the requesting code was written.

There's lots of places where crashing isn't actually what you want, and you have to capture all errors, explicitly log it, and then move on... But when you can, checking for success or success and a handful of expected and recoverable errors is very nice.

Re: Don't “let it crash”, let it heal

#27

Post by rtpg »

praptak wrote: post #19

A condition that "should not happen" might still be a problem specific to a particular request. If you "just crash" it turns this request from one that only triggers a http 500 response to one that crashes the process. This increases the risk of Query of Death scenarios where the frontend that needs to serve this particular request starts retrying it with different backends and triggers restarts faster than the proce…

My impression is that in Erlang land each process handler is really cheap so you can just keep on showing up with process handlers and not reach exhaustion like you do with other systems (at least in pre-async worlds...)

Re: Don't “let it crash”, let it heal

#28

Post by zwnow »

knome wrote: post #24
zwnow wrote: post #20

Earlier quoted context omitted.

This is funny given Elixir/Erlangs whole idea is "let it crash". In Go I just have a Recovery Middleware for any type of problem. Don't know how other langs do it tho

erlang doesn't crash the program, it crashes the thread. erlang has a layered management system built in as part of OTP (open telecom platform, erlang was built for running highly concurrent telephony hardware). when a thread crashes, it dies and signals its parent. the parent then decides what to do. usually, that's just restarting the worker. maybe if ten workers have crashed in a minute, the manager itself will di…

It's not a misconception given that Elixir Forum and its Discords members will say that to you. Also I never assumed the whole program crashed so why would you explain this to me? Why would one Blog guy know it better than a lot of other Elixir devs?

Re: Don't “let it crash”, let it heal

#29

Post by kimi »

ramchip wrote: post #12
HexDecOctBin wrote: post #9

How does restarting the process fix the crash? If the process crashed because a file was missing, it will still be missing when the process is restarted. Is an infinite crash-loop considered success in Erlang?

I recommend https://ferd.ca/the-zen-of-erlang.html starting from "if my configuration file is corrupted, restarting won't fix anything". The tl;dr is it helps with transient bugs.

...and does no harm for unfixable bugs. It's the logical equivalent of "switch off and on again" that as we know fixes most issues by itself, but happening only on a part of your software deployment, so most of it will keep running.

Re: Don't “let it crash”, let it heal

#30

Post by nine_k »

corysama wrote: post #16
HexDecOctBin wrote: post #9

How does restarting the process fix the crash? If the process crashed because a file was missing, it will still be missing when the process is restarted. Is an infinite crash-loop considered success in Erlang?

I’m only an armchair expert on Erlang. But, having looked into it repeatedly for a couple decades, my take-away is the “Let it crash” slogan is good. But, also presented a bit out of context. Or, at least assuming context that most people don’t have. Erlang is used in situations involving a zillion incoming requests. If an individual request fails… Maybe it was important. Maybe it wasn’t. If it was important, it’s ex…

Let it crash, so that if something goes wrong, it does not do so silently.

Let it crash, because a relevant manager will detect it, report it, clean it up, and restart it, without you having to write a line of code for that.

Let it crash as soon as possible, so that any problem (like a crash loop) is readily visible. It's very easy to replace arbitrary bits of Erlang code in a running system, without affecting the rest of it. "Fix it in prod" is better than "miss it in prod", especially when you cannot stop the prod ever.

Post reply on HN