I saw something similar back in 2010 for java but it was marketed as a different product. It was some kind of a debugger which would let me forward and rewind time and the state of the program with a single slider. It was marvelous and totally unusable for the monster java EE stuff we were working on but sounded awesome for sane, smaller projects.
When "letting it crash" is not enough
21–30 of 84 posts
Re: When "letting it crash" is not enough
#22The approach of check-pointing computation such that it is restartable sounds similar to a time-traveling debugger, like rr or WinDbg: https://rr-project.org/ https://learn.microsoft.com/windows-hardware/drivers/debugge... Some Googling found Checkpoint/Restore In Userspace, or CRIU. It’s like Flawless, but for Linux processes: https://criu.org/Main_Page I bet that Flawless can make better guarantees about reliabilit…
Our time travel debugger at undo.io also uses a similar approach.
But the world is different for time travel debug, in that you can replay the whole history of the recorded execution (my understanding is that this is not required for Durable Execution) but you cannot resume the real process.
In fact, since it's used for capturing faults, you wouldn't generally want to resume execution - it's going to fail the same way (whereas a higher level restart framework might allow you to throw away some bad state and continue the computation - at the cost of not being able to precisely duplicate the bug).
Re: When "letting it crash" is not enough
#23I thought the idea of "exactly once" was a very questionable claim given what we know about computing, specially distributed. Then somewhere else you see this gem. > Workflows in flawless are written in Rust, in fact they are just regular Rust functions. This means that they can contain arbitrary logic. But instead of native code, the functions are compiled to WebAssembly and executed in a completely deterministic en…
The main page ( https://flawless.dev/ ) has a diagram/video that shows how it would work and it basically writes any 'side effects' to a log which is used to track their existence. If they exist in the log, you read them; if they don't you run the code that generates them and then log it. It's interesting, but I can't imagine how it is going to work at scale, both in terms of managing state across a very large applic…
As the saying goes, "it is turtles all the way down". How do you ensure the log is "written" and "synced" _exactly once_?
Re: When "letting it crash" is not enough
#24Erlang and OTP, as ugly as it is (to me) got a lot right on this front already.
Re: When "letting it crash" is not enough
#25As it was mentioned, this sounds very much like time-traveling debuggers like RR, which also records such database.
But I don't exactly understand: If you exactly recover the same state, don't you end up with the same faulty undesirable state? (That's what you also get with RR, by intention, to debug the problem.)
Clearly, here this is not the intention, i.e. you don't want to recover exactly the same state. So, it means, some parts will not be recovered. How would it be decided what parts to recover and what parts not? This seems like an impossible problem to solve in general.
And then, the state will not be exactly the same. It might be now in a sane state, but is it the right state that the user wants?
Re: When "letting it crash" is not enough
#26> This brings me to a recent discovery I made, another approach to dealing with failure that completely blew my mind . It's commonly known under the name durable execution, and is so new that most developers never have heard of it. Some feedback: this paragraph lands like a Reddit post from a teenager, who just took a big drag off a joint, and thinks they've invented a new field of science. (And also hasn't completed…
They're not claming that they invented it, it's an existing term. This blogpost lists 14 different projects in the space: https://www.golem.cloud/post/the-emerging-landscape-of-durab...
Re: When "letting it crash" is not enough
#27As it was mentioned, this sounds very much like time-traveling debuggers like RR, which also records such database. But I don't exactly understand: If you exactly recover the same state, don't you end up with the same faulty undesirable state? (That's what you also get with RR, by intention, to debug the problem.) Clearly, here this is not the intention, i.e. you don't want to recover exactly the same state. So, it m…
I guess it insulates you against bugs in the VM implementation, plus against (transient) failures of the host system.
Re: When "letting it crash" is not enough
#28Two big question marks after reading this and the linked home page (partially pointed out in other comments):
- If there's a flaw in your application code that causes a crash (as is the motivating example in the essay), then restoring the entire program into the state it was in just before the crash happened would just cause it to crash again ad infinitum. Sure, this model helps against "my VM instance got preempted", but that's a pretty different category of "crash" (and also notably unrelated to supervision trees).
- "External" state (like an API endpoint being down/returning gibberish) can be part of the reason why your program got into a bad state. In fact, that's disproportionately likely, since external "weirdness" is comparatively hard to cover exhaustively in tests. In such a situation, the suggested computation model would never be able to recover even when restarted, because it would forever retain the (bad) API response. Effectively, this is just caching all the non-pure effects of your computation, and we all know about cache invalidation being a hard problem...
Re: When "letting it crash" is not enough
#29The approach of check-pointing computation such that it is restartable sounds similar to a time-traveling debugger, like rr or WinDbg: https://rr-project.org/ https://learn.microsoft.com/windows-hardware/drivers/debugge... Some Googling found Checkpoint/Restore In Userspace, or CRIU. It’s like Flawless, but for Linux processes: https://criu.org/Main_Page I bet that Flawless can make better guarantees about reliabilit…
Docker[0] supports CRIU, but it has always been experimental. I always assumed it would be a valuable tool for test cases, but I never used it.
[0]: https://docs.docker.com/engine/reference/commandline/checkpo...
Re: When "letting it crash" is not enough
#30I wonder if this would end up with a bloated log. Imagine that you download a large file and then count the occurrences of a word. You wouldn't want to save that whole file to the log, after you count occurrences you don't need it, but it must stay in the log incase the execution is retried.
Maybe you'd want a wrapper around certain sections to say: only only log the output of this group of statements.