Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When "letting it crash" is not enough

flawless.dev

Post reply on HN
Print view View original

41–50 of 84 posts

Re: When "letting it crash" is not enough

#41

Post by yawboakye »

properly examined, this is another form of compilation, except slightly handicapped. i wonder if it would be possible to unroll/compile the function three-address code[0] style before execution, instead of the current model where you use specialized apis (eg temporal[1] and flawless[2]). i want to write my code like any program should be written, without special incantations, but still get durable execution as a non-negotiable benefit.

[0]: https://en.wikipedia.org/wiki/Three-address_code

[1]: https://temporal.io/

[2]: https://flawless.dev/

Re: When "letting it crash" is not enough

#43

Post by bkolobara »

baby wrote: post #39

Big problem with this approach is doing the same thing twice, which is a big no no in a lot of applications. For example, if you do something, then crash before you get the chance to update your durable execution object, then you’ll do it again after restarting. We were dealing with such code in a TCB and the only way is to audit your code and ensure that any action that can only be done once at most saves that actio…

Hi! I'm the author of flawless.

Exactly once execution is one of the guarantees I want to provide. Flawless should always give you peace of mind, and in cases it can't guarantee exactly once execution it will sacrifice progress. When interacting with the real world, hardware and software can fail in creative ways and it's not always possible to automatically recover without manual intervention. Sometimes it's just not possible to solve issues without looping in a human. But flawless will not repeat a call if it can't guarantee it was not performed or if it's not idempotent.

For many features, like HTTP requests, flawless uses a dual commit system. In cases where data was sent to an external server, but no response is received (timed out), we can't know what the external system observed, and will not allow progress to continue (fail the workflow). You can relax this requirement by marking the HTTP call as idempotent.

Re: When "letting it crash" is not enough

#45

Post by steveBK123 »

> The more complex your application is, the more variables you need to keep track of everything. Eventually it becomes impossible for developers to predict all combinations of state that these variables will form.

I discovered an interesting simple case of this on my utility's websites. Autopay for some reason didn't trigger. It showed I had autopay enabled, but no payment ever made.

Tried to pay manually, got "error: autopay enabled, no one-time payment allowed". The only way to pay my bill was to delete my autopay, and manually pay.

I now cannot re-enable autopay getting the error "autopay disabled for this account" (possibly because it had the past-due balance I had just paid?).

Crossing my fingers if I log back in next week all the states will reconcile themselves. Software!

Re: When "letting it crash" is not enough

#48

Post by pjc50 »

MauranKilom wrote: post #28

Two big question marks after reading this and the linked home page (partially pointed out in other comments): - If there's a flaw in your application code that causes a crash (as is the motivating example in the essay), then restoring the entire program into the state it was in just before the crash happened would just cause it to crash again ad infinitum. Sure, this model helps against "my VM instance got preempted"…

Original article is now ironically crashed, but: my last job was working on a point-of-sale system which used this kind of append-only transaction system and a "crash and reboot on failure" model. Every button press got turned into one or more transactions. This had the nice property that if something failed, most of the time it was before anything got written, so the system would reboot and leave you back on the screen before your last button press. The state could also be shipped over to the developer's PC so you could repro that state under the debugger. There had to be a "detach account" getout clause for cases where the sequence of transactions caused a crash on load, which was rare but possible.

The hardest part was of course managing external state and journaling exactly where you had got to with external transaction APIs. Further backend reconciliation was available to flag this (and avoid Post Office scenarios).

Note that French NF525 almost mandates this design, at least for point-of-sale systems: every financial transaction has to be durably written for tax auditing purposes.

Re: When "letting it crash" is not enough

#50

Post by torginus »

bkolobara wrote: post #38
MauranKilom wrote: post #28

Two big question marks after reading this and the linked home page (partially pointed out in other comments): - If there's a flaw in your application code that causes a crash (as is the motivating example in the essay), then restoring the entire program into the state it was in just before the crash happened would just cause it to crash again ad infinitum. Sure, this model helps against "my VM instance got preempted"…

Hi! I'm the author of the essay. Durable execution is meant to complement your application. You will never want to model everything with it. It solves the problem of needing to decide how often to manually make snapshots of some important state, this becomes implicit. Workflows in flawless can still fail, you could call the `panic` function, or divide by zero. In the end it's arbitrary compute. "External" state is on…

I feel like this approach might still pose some challenges or issues with regards to time or stale data. A couple of problematic scenarios:

- Application requests a JWT token. It then crashes and gets restarted. It gets past the problematic point, but later when trying to make a request, it crashes due to the cached token being expired.

- Application interacts with the current time in a meaningful manner. Due to the log replay, it will always live in the past and when switching from the cache-sourced time to the current time, some issues might occur, like deltas being larger than expected

- Application goes through a webshop checkout flow. After restart, some of the items in its cart have been already sold, but the app doesn't check this, since it already went through a (cached) check and got the result

Post reply on HN