it applies to human civilization as well....
When "letting it crash" is not enough
51–60 of 84 posts
Re: When "letting it crash" is not enough
#52> .. durable execution, and is so new that most developers never have heard of it. It's called checkpoint/restart, and was a feature of some early operating systems. Mostly for programs whose run time exceeded the mean time before failure of the system. Tandem's whole system concept was built around that. Amusingly, Second Life, of all things, has durable execution of the little LSL programs that make in-world object…
There's a decent chance checkpoint/restart has been with us since the days of paper tape.
It's moderately amusing to see "save state somewhere" described as so new people haven't heard of it.
Re: When "letting it crash" is not enough
#53Big problem with this approach is doing the same thing twice, which is a big no no in a lot of applications. For example, if you do something, then crash before you get the chance to update your durable execution object, then you’ll do it again after restarting. We were dealing with such code in a TCB and the only way is to audit your code and ensure that any action that can only be done once at most saves that actio…
Hi! I'm the author of flawless. Exactly once execution is one of the guarantees I want to provide. Flawless should always give you peace of mind, and in cases it can't guarantee exactly once execution it will sacrifice progress. When interacting with the real world, hardware and software can fail in creative ways and it's not always possible to automatically recover without manual intervention. Sometimes it's just no…
I'm not sure exactly once can be done in the presence of failure. When you come back online, if you're not sure whether a call happened or not, you either drop it or try again - and that gives you at most or at least once.
Re: When "letting it crash" is not enough
#54Earlier quoted context omitted.
If your system provides strong consistency, it is possible to build exactly once processing over it. NFSv3 is an existence proof, and there are plenty of theoretical results showing it is possible. Roughly speaking, you run the job and install the result in the consistent store iff your output register is null. If you side effect outside of the exactly once system, and the receiver can’t suppress duplicate notificati…
> If you side effect outside of the exactly once system, and the receiver can’t suppress duplicate notifications then you’re screwed If you are getting duplicate notifications, then you don't have an "exactly once" system, but definition. Exactly once systems are not possible, bu your own explanation. You can store data in a durable way such that applying it multiple times is safe, but that is not "exactly once". You…
Many, many distributed systems provide exactly once semantics by giving up availability.
They don’t compose well with systems that don’t provide exactly once semantics, but that doesn’t somehow make them stop existing or stop working.
Also, you don’t need to arrange for data structures that can be modified multiple times safety.
You just need the exactly once system to support transactions, compare and swap, atomic rename, etc. Such things are usually pretty easy to retrofit.
For instance, Google and Microsoft S3 support conditional writes that check the etag for equality. That’s enough to let you layer consistency and exactly once on top. AWS S3 doesn’t support this, and people have been asking for it for a long time.
These things are extremely well understood and done at scale all the time.
Edit: I think they use etags for this. They might expose it as atomic rename or create if not exists instead. The expressive power of the two should be equivalent.
Re: When "letting it crash" is not enough
#55Two big question marks after reading this and the linked home page (partially pointed out in other comments): - If there's a flaw in your application code that causes a crash (as is the motivating example in the essay), then restoring the entire program into the state it was in just before the crash happened would just cause it to crash again ad infinitum. Sure, this model helps against "my VM instance got preempted"…
> cache invalidation being a hard problem
For those unaware: "There are only two hard things in computer science: cache invalidation and naming things." Plus variants like, "Oh, and off-by-one errors."
Re: When "letting it crash" is not enough
#56Erlang and OTP, as ugly as it is (to me) got a lot right on this front already.
Indeed. I do wish there were a few more "extras" in OTP, because "let it crash" needs some more details in some circumstances.
For instance if you have a system with a user interface and some various components, like say, a database, and the database becomes unavailable, you don't want the entire system to crash. You want it to display an error message to the user and maybe go into some kind of diagnostic mode or other "things are not normal" state.
Something like https://github.com/jlouis/fuse is one approach. I had a small stab at creating something similar: https://github.com/davidw/hardcore
Re: When "letting it crash" is not enough
#57The approach of check-pointing computation such that it is restartable sounds similar to a time-traveling debugger, like rr or WinDbg: https://rr-project.org/ https://learn.microsoft.com/windows-hardware/drivers/debugge... Some Googling found Checkpoint/Restore In Userspace, or CRIU. It’s like Flawless, but for Linux processes: https://criu.org/Main_Page I bet that Flawless can make better guarantees about reliabilit…
I saw a tech demo somewhere that showcased that Qt applications on wayland could be checkpointed and restored with CRIU.
Re: When "letting it crash" is not enough
#58Earlier quoted context omitted.
The main page ( https://flawless.dev/ ) has a diagram/video that shows how it would work and it basically writes any 'side effects' to a log which is used to track their existence. If they exist in the log, you read them; if they don't you run the code that generates them and then log it. It's interesting, but I can't imagine how it is going to work at scale, both in terms of managing state across a very large applic…
As the saying goes, "it is turtles all the way down". How do you ensure the log is "written" and "synced" _exactly once_?
Just write the action of writing a log to a different log file. Duh. /s
Re: When "letting it crash" is not enough
#59Big problem with this approach is doing the same thing twice, which is a big no no in a lot of applications. For example, if you do something, then crash before you get the chance to update your durable execution object, then you’ll do it again after restarting. We were dealing with such code in a TCB and the only way is to audit your code and ensure that any action that can only be done once at most saves that actio…
Hi! I'm the author of flawless. Exactly once execution is one of the guarantees I want to provide. Flawless should always give you peace of mind, and in cases it can't guarantee exactly once execution it will sacrifice progress. When interacting with the real world, hardware and software can fail in creative ways and it's not always possible to automatically recover without manual intervention. Sometimes it's just no…
If your system refuses to execute an action that it can’t guarantee didn’t occur, isn’t that “at most once” semantics?
Re: When "letting it crash" is not enough
#60A bit of an aside, but I really wish the animation on the flawless.dev homepage "crashed" at a non-ideal spot.
As it is, the animation crashes at the most opportune moment possible: after a side-effect statement has been persisted to the log, and even before the subsequent statement executes.
Where I want to see that animation crash is in the middle of that "HTTP request" at the bottom, where that request is an HTTP POST, and where the crash is "the server successfully received the request, and processed it, but we crashed before we could get the response."
The resuming execution's log is either "empty" … or "we sent it, but never got the result" — the former has you repeating the side-effect, and the in latter, you can at least sort of tell you're doomed.