Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When "letting it crash" is not enough

flawless.dev

Post reply on HN
Print view View original

11–20 of 84 posts

Re: When "letting it crash" is not enough

#11

Post by vinkelhake »

I implemented something similar at work for a phone handling system. We had a "workflow" system that let clients script how calls should be handled. All side effects that the script was allowed to make happened through an API with functionality like reading DTMF input, playing media, adding another call leg etc.

Like the article says, we would record a log of all the interactions with the outside world. The lower level parts of the system had the capability of transferring an in-progress call from one server to another. SIP and RTP traffic would be moved to another server, and finally the workflow log was sent. The server that took over the call would then replay the script and arrive at the same execution state that the first server had before the call was moved. Workflow authors didn't have to care about any of this. To them it looked like the call started and ended on the same server.

Re: When "letting it crash" is not enough

#12

Post by Animats »

> .. durable execution, and is so new that most developers never have heard of it.

It's called checkpoint/restart, and was a feature of some early operating systems. Mostly for programs whose run time exceeded the mean time before failure of the system.

Tandem's whole system concept was built around that.

Amusingly, Second Life, of all things, has durable execution of the little LSL programs that make in-world objects go. They're checkpointed every minute or two, and if a region crashes, they are restarted, stack, heap, and all. They survive machine crashes and ports to new hardware. Even migration from a dedicated data center to AWS. Some have been running for well over a decade. Internally, they are Mono programs.

Re: When "letting it crash" is not enough

#13

Post by orbital-decay »

Durable execution and state restoration is also a great way to get close to the crashing point again, though.

The entire point of crashing early is to reset the state of the program completely, hoping that the path that crashes it will not be taken too often. (and it only works well in the situations where crashing means negligible interruption of the service) By preserving the state between crashes, you diminish the pool of possible non-crashing states.

Re: When "letting it crash" is not enough

#15

Post by sausagefeet »

hedora wrote: post #7
omeid2 wrote: post #2

I thought the idea of "exactly once" was a very questionable claim given what we know about computing, specially distributed. Then somewhere else you see this gem. > Workflows in flawless are written in Rust, in fact they are just regular Rust functions. This means that they can contain arbitrary logic. But instead of native code, the functions are compiled to WebAssembly and executed in a completely deterministic en…

Exactly Once is basically a solved problem. The CAP theorem says you can either get consistency or availability in a system that has network partitions. If you give up consistency, you end up corrupting data every once in a while. If you don’t, then you can have exactly once, but it might take a long time if there’s a network failure. NFSv3 solved this back in the 1980’s. (V2 and V1 may have, but I don’t know.) It di…

This comment has some errors in it:

1. Exactly Once is absolutely not a solved problem. CAP says nothing about 'exactly once', it's about design choices for your data. But you can do all sorts of side-effectful things when it comes setting the data.

2. Giving up consistency does not mean you "corrupt your data every once in awhile". I don't know why you would say that. Choosing availability means you have to make decisions about how your data becomes consistent, but nothing about that means it is corrupted.

3. Choosing consistency says nothing about "exactly once". Consider this basic workflow backed by a consistent database: request comes into service, service sends email, services goes to store that email was sent in consistent database and crashes, user gets crash back, reruns request, email sent again. Oh so send the email AFTER it's stored in the database, well service crashes after saving in database, same problem.

Re: When "letting it crash" is not enough

#18

Post by kotborealis »

MarkSweep wrote: post #3

The approach of check-pointing computation such that it is restartable sounds similar to a time-traveling debugger, like rr or WinDbg: https://rr-project.org/ https://learn.microsoft.com/windows-hardware/drivers/debugge... Some Googling found Checkpoint/Restore In Userspace, or CRIU. It’s like Flawless, but for Linux processes: https://criu.org/Main_Page I bet that Flawless can make better guarantees about reliabilit…

There's also an implementation of time-traveling debugging in Qenu[0], which sounds really nice but does not actually work reliably on larger use-cases.

[0] https://www.linux-kvm.org/images/d/d0/02x06b-DeterministicRe...

Re: When "letting it crash" is not enough

#19

Post by bjconlan »

solatic wrote: post #6

Flawless sounds a lot like https://temporal.io/ . I'm wondering if it has the same scalability concerns - sticking everything in Postgres is fine at small-ish scale, but what happens when you outgrow Postgres, either because you have higher availability requirements (can't handle primary DB restarts) or because of the sheer volume of the workload?

Yeah the whole thing reminded me of the blog post they talk about regarding a time travelling debugger...

https://temporal.io/blog/time-travel-debugging-production-co...

Anyways I wish all the best for flawless as this problem is worth solving/popularizing.

Re: When "letting it crash" is not enough

#20

Post by notfed »

> This brings me to a recent discovery I made, another approach to dealing with failure that completely blew my mind . It's commonly known under the name durable execution, and is so new that most developers never have heard of it.

Some feedback: this paragraph lands like a Reddit post from a teenager, who just took a big drag off a joint, and thinks they've invented a new field of science. (And also hasn't completed it so isn't ready to share it yet!)

Of course, everything's already been thought of. What you've described so far as durable computing sounds indistinguishable from event sourcing.

The code snippet is intriguing, though. We're abstracting event sourcing away from the developer---it happens automatically for every non-purely-functional operation---so the programmer can just write normal code.

Though, you'd have a lot of events, and I'd wonder about performance and scalability for larger apps.

Also, assuming events are at the finest level of granularity, I also imagine this starts to look like the equivalent of some kind of intermittent heap dump.

Post reply on HN