Live data from Hacker News

When "letting it crash" is not enough

flawless.dev

61–70 of 84 posts

Re: When "letting it crash" is not enough

#61

The approach of check-pointing computation such that it is restartable sounds similar to a time-traveling debugger, like rr or WinDbg: https://rr-project.org/ https://learn.microsoft.com/windows-hardware/drivers/debugge... Some Googling found Checkpoint/Restore In Userspace, or CRIU. It’s like Flawless, but for Linux processes: https://criu.org/Main_Page I bet that Flawless can make better guarantees about reliabilit…

I have been looking for an excuse to use RR, a problem that would have been solved much easier if I had it. I haven't yet found one.

Does anyone here have experience with it? Maybe know the sort of problems RR excels at helping find?

EDIT: And is there some way to still use a better visual overlay a-la Visual Studio or QTCreator?

Re: When "letting it crash" is not enough

#62
post #56

Erlang and OTP, as ugly as it is (to me) got a lot right on this front already.

Indeed. I do wish there were a few more "extras" in OTP, because "let it crash" needs some more details in some circumstances. For instance if you have a system with a user interface and some various components, like say, a database, and the database becomes unavailable, you don't want the entire system to crash. You want it to display an error message to the user and maybe go into some kind of diagnostic mode or oth…

Let it crash is just the start of the process, really. Write the happy path, with pattern matching so that if things failed, it does crash, like ok = do_the_thing(). You're also expected to monitor crash reports and (live) update the system to handle crashes that need more specific handling than a simple restart.

In a distributed system you always need to handle the case where you sent a request and didn't get a response and the request may or may not have been processed. Once you accept that, it's totally reasonable for the request processor to crash during a request, because the requester always needs to be know what to do in case they get no response or a non-specific error.

Re: When "letting it crash" is not enough

#63

A bit of an aside, but I really wish the animation on the flawless.dev homepage "crashed" at a non-ideal spot. As it is, the animation crashes at the most opportune moment possible: after a side-effect statement has been persisted to the log, and even before the subsequent statement executes. Where I want to see that animation crash is in the middle of that "HTTP request" at the bottom, where that request is an HTTP…

In this scenario, the workflow is marked as "failed" and requires manual approval to continue. Sometimes you just can't resolve the issue without human input.

Re: When "letting it crash" is not enough

#64

Two big question marks after reading this and the linked home page (partially pointed out in other comments): - If there's a flaw in your application code that causes a crash (as is the motivating example in the essay), then restoring the entire program into the state it was in just before the crash happened would just cause it to crash again ad infinitum. Sure, this model helps against "my VM instance got preempted"…

> then restoring the entire program into the state it was in just before the crash happened would just cause it to crash again ad infinitum.

Hopefully!

But, sadly, this is not always the case. It's sad because it's incredibly hard to debug crashes that don't happen deterministically.

Re: When "letting it crash" is not enough

#65

Two big question marks after reading this and the linked home page (partially pointed out in other comments): - If there's a flaw in your application code that causes a crash (as is the motivating example in the essay), then restoring the entire program into the state it was in just before the crash happened would just cause it to crash again ad infinitum. Sure, this model helps against "my VM instance got preempted"…

Hi! I'm the author of the essay. Durable execution is meant to complement your application. You will never want to model everything with it. It solves the problem of needing to decide how often to manually make snapshots of some important state, this becomes implicit. Workflows in flawless can still fail, you could call the `panic` function, or divide by zero. In the end it's arbitrary compute. "External" state is on…

Funnily enough, this is actually a massive problem when working with cloud automation APIs. Terraform and the like kinda handle this problem by calculating / storing the “goal state” and then looking at the system’s current state, and coming up with a “plan” to reconcile it.

Unfortunately, cloud provider APIs are usually eventually consistent, and getting a full snapshot at scale is nigh impossible.

So, in order to work around this, I effectively built a write ahead log style atop Postgres. Something like Sagas would have been great, but as far as I can tell, there was no real pattern for multiple Sagas operating on global state doing coordination. This is where Postgres SSI came in handy, where I could read the assumed state of the system, and if another worker came in and manipulated it, the write ahead entry wouldn’t get written as the txn would fail to commit. The write ahead entry would then get asynchronously processed by another worker, in case the first worker failed.

Re: When "letting it crash" is not enough

#66
IMHO the beauty of "let it crash" is that you can code very tersely while maintaining data safety. In order for "let it crash" to work you need two things:

1. A tech stack that isolates crashes such that they do not affect the rest of the system. Example: you receive a malformed API response. The code responsible for parsing it crashes, but the rest of your application does not.

2. You use this to code in a declarative style.

Coding declaratively means your code has the shape of the data you expect to have.

Example (in elixir):

Let's say you are calling an external API where you expect to get back a list with a single element.

If you code non decleratively, your code might look like this:

  thing_i_want = List.first(my_api_response)
Now imagine that for some reason the external API sent you a list with two elements. That code still runs! It doesn't crash, but you are now in a strange state. The data you are passing into the system is unexpected.

Coding declaratively you would write:

    [thing_i_want] = my_api_response
In that case, if for whatever reason the external API sends you a list with more than one element the code will crash. In BEAM languages that is fine--the rest of the application should be ok, and you should log why you crashed and when, so you can look into it, but you are not ingesting bad data into the system.

The alternative is to code defensively: check the lenght of the list before extracting the first/only element. That works, but it tends to be very verbose and more importantly it means the entire call stack needs to be aware that you might get exceptions bubbled up from anywhwere.

Of course, for this to work you need software that has lightweight processes and a supervisor architecture..not usually worth it unless you're getting it for free! (as with elixir/erlang).

Re: When "letting it crash" is not enough

#67
post #54

Earlier quoted context omitted.

> If you side effect outside of the exactly once system, and the receiver can’t suppress duplicate notifications then you’re screwed If you are getting duplicate notifications, then you don't have an "exactly once" system, but definition. Exactly once systems are not possible, bu your own explanation. You can store data in a durable way such that applying it multiple times is safe, but that is not "exactly once". You…

Many, many distributed systems provide exactly once semantics by giving up availability. They don’t compose well with systems that don’t provide exactly once semantics, but that doesn’t somehow make them stop existing or stop working. Also, you don’t need to arrange for data structures that can be modified multiple times safety. You just need the exactly once system to support transactions, compare and swap, atomic r…

Exactly once is a mathematical impossibility. It has been proven. See the two generals problem. You can get similar end results if you have idempotent calls and allow duplicate calls to be made. This gives the illusion or impression of exactly once.

Re: When "letting it crash" is not enough

#68
post #64

Two big question marks after reading this and the linked home page (partially pointed out in other comments): - If there's a flaw in your application code that causes a crash (as is the motivating example in the essay), then restoring the entire program into the state it was in just before the crash happened would just cause it to crash again ad infinitum. Sure, this model helps against "my VM instance got preempted"…

> then restoring the entire program into the state it was in just before the crash happened would just cause it to crash again ad infinitum. Hopefully! But, sadly, this is not always the case. It's sad because it's incredibly hard to debug crashes that don't happen deterministically.

[deleted]

Re: When "letting it crash" is not enough

#69
post #12

> .. durable execution, and is so new that most developers never have heard of it. It's called checkpoint/restart, and was a feature of some early operating systems. Mostly for programs whose run time exceeded the mean time before failure of the system. Tandem's whole system concept was built around that. Amusingly, Second Life, of all things, has durable execution of the little LSL programs that make in-world object…

There's a decent chance checkpoint/restart has been with us since the days of paper tape. It's moderately amusing to see "save state somewhere" described as so new people haven't heard of it.

I remember there was one for unix back in the 90's, I think called Condor, that could be used to migrate long running processes to other machines. (I think the tricky part was restoring external connections like file handles.)

And before that, there were TeX and emacs, which do a memory dump and restore to save time loading their initial state. One of those, I think emacs, would tweak a core dump file to be a valid executable again.

Re: When "letting it crash" is not enough

#70
post #12

> .. durable execution, and is so new that most developers never have heard of it. It's called checkpoint/restart, and was a feature of some early operating systems. Mostly for programs whose run time exceeded the mean time before failure of the system. Tandem's whole system concept was built around that. Amusingly, Second Life, of all things, has durable execution of the little LSL programs that make in-world object…

There's a decent chance checkpoint/restart has been with us since the days of paper tape. It's moderately amusing to see "save state somewhere" described as so new people haven't heard of it.

Checkpoint/Restore I feel is a bigger concept than just saving state. At the zeroth level it's a system that can correctly stop and serialize a running process (as criu https://github.com/checkpoint-restore/criu has shown is a huge pain in the ass to still not be perfect) in a way that can initiated from within the process itself.

The 1st level more-work-but-easier way to do this is to build or use a heavily constrained VM/language you run from within your main application that doesn't allow for most of the hard problems to even exist.

I can't find any ready-made tools to do this that I wouldn't consider an endeavor. Emacs has to be the most famous application to utilize dump/restore state but it's not exactly turnkey.

Post reply on HN