Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Durable execution should be lightweight

dbos.dev

Post reply on HN
Print view View original

31–40 of 45 posts

Re: Durable execution should be lightweight

#31

Post by KraftyOne »

ethbr1 wrote: post #21
KraftyOne wrote: post #17

Earlier quoted context omitted.

That's exactly what this model is! The @Step decorator is for external state modifications. Then @Workflows orchestrate steps. The example shows the simplest possible external state modification--a print to the terminal. Steps can be tried multiple times (if a failure happens mid-step) but never re-execute once complete. Since idempotency can't be added externally, that's the strongest possible guarantee any orchestr…

You're forcing adopters to divide any state-impactful activity into its own function (because only functions can be decorated with step, no?). That's seriously inelegant when scaled to larger codebases. Regional tagging (e.g. safe/unsafe) would be a better approach, as it would allow developers to more naturally protect code, without redefining its structure to suit your library. You start to grok the problem here, b…

Yeah, this definitely requires splitting state-impactful activity into its own function. That's good practice anyways, though I understand it might be a pain in large codebases. Regional tags are definitely an interesting alternative!

For the UI example, I don't think you'd use durable execution for most of the UI--it's just not needed. But maybe there's one button that launches a complex asynchronous background task, and you'd use durable execution for that (with careful workflow ID management to ensure idempotency and allow you to retrieve the status of the background task).

Re: Durable execution should be lightweight

#32

Post by no_wizard »

peterkelly wrote: post #8

Durable execution is best done at the level of a language implementation, not as a library. A workflow engine I recently built provided an interpreter for a Scheme-based language that, for each blocking operation, took a snapshot of the interpreter state (heap + stack) and persisted that to a database. Each time an operation completes (which could be after hours/days/weeks), the interpreter state is restored from the…

Effectively you’re using the scheme language in question as a state machine?

xstate has a similar mechanism for restoring state from defined state schema which on the surface seems similar

Re: Durable execution should be lightweight

#33

Post by cle »

peterkelly wrote: post #8

Durable execution is best done at the level of a language implementation, not as a library. A workflow engine I recently built provided an interpreter for a Scheme-based language that, for each blocking operation, took a snapshot of the interpreter state (heap + stack) and persisted that to a database. Each time an operation completes (which could be after hours/days/weeks), the interpreter state is restored from the…

A contributor in this space that I always thought was under-appreciated is Amazon SWF + Flow Framework. It's an older technology and SWF itself is deprecated, but it is a weird middle ground here in that it makes language-level modifications to inject workflow state persistence and coordination into Java code (via AspectJ). The original concept was to build a "distributed CPU" (ex see https://docs.aws.amazon.com/amazonswf/latest/awsflowguide/aw...).

SWF was the predecessor for AWS Step Functions. IIUC the lesson learned from SWF was that it was just too flexible, and imposing a more limiting set of constraints was both easier for the programmer to use & reason about, and made for simpler and faster execution.

Re: Durable execution should be lightweight

#34

Post by CMCDragonkai »

KraftyOne wrote: post #18
martinpeck wrote: post #3

I think the example given in this blog post might need a "health warning" that steps should, generally, be doing more than just printing "hello". I can imagine that the reads and writes to Postgres for a large number of workflows, each with a large number of small steps called in a tight loop, would cause some significant performance problems. The examples given on their main site are a little more meaningful.

Yes, that's totally fair. Usually, a step is a meaningful unit of work, such as a API call that performs an external state modification. Because each step is a fair chunk of work, and the overhead is just one write per step, this scales well in practice--as well as Postgres scales, up to 10K+ operations/second.

I feel like you can generalise this to any transactional key value system, which can scale better.

Re: Durable execution should be lightweight

#35

Post by tenken »

I have never used it, but a predasessor of mine talked about Clipper alot and I believe it allowed remote execution blocks tied to a storage backend, in this case I'm talking about xBase languages ...

I think also Rebol supports remote execution blocks ...

Re: Durable execution should be lightweight

#36

Post by dalyons »

martinpeck wrote: post #3

I think the example given in this blog post might need a "health warning" that steps should, generally, be doing more than just printing "hello". I can imagine that the reads and writes to Postgres for a large number of workflows, each with a large number of small steps called in a tight loop, would cause some significant performance problems. The examples given on their main site are a little more meaningful.

Your intuition is good - having worked with something similar to this, it works great but does not scale very well. The step journaling is pretty brutal to postgres/rdbms , and you hit vertical scaling limits quicker than you would like

Re: Durable execution should be lightweight

#37

Post by shipp02 »

KraftyOne wrote: post #20
Diggsey wrote: post #5

Earlier quoted context omitted.

The example is overly simplified. It glosses over many of the subtle-but-important aspects of durable execution. For example: - Steps should be small but fallible operations - eg. sending a request to an external service. You generally want to tailor the retry logic on steps to the specific task they are doing. Doing too much in a step can increase failure rates or cause other problems due to the at-least-once behavi…

This is a great answer, and yes, those are critical aspects of durable execution. Maybe I should write a follow-on post that goes into more detail...

I'd love to read it. Getting exactly once semantics is quite an interesting topic.

Re: Durable execution should be lightweight

#38

Post by shipp02 »

KraftyOne wrote: post #19
shipp02 wrote: post #4

This seems like temporal only without as much server and complexity. Maybe they ignore it or it really is that simple. Overall really cool! There are some scalability concerns that are brought that I think are valid but maybe you have a Postgres server backing up every few servers that need this kind of execution. Also, every function shouldn't be its own step but needs to be divided into larger chunks where every re…

Thanks! DBOS is simpler not because it ignores complexity, but because it uses Postgres to deal with complexity. And Postgres is a very powerful tool for building reliable systems!

Have you had scalability issues because your tables got too big?

Is there a mechanism to GC workflows that are completed?

Re: Durable execution should be lightweight

#39

Post by KraftyOne »

shipp02 wrote: post #38
KraftyOne wrote: post #19

Earlier quoted context omitted.

Thanks! DBOS is simpler not because it ignores complexity, but because it uses Postgres to deal with complexity. And Postgres is a very powerful tool for building reliable systems!

Have you had scalability issues because your tables got too big? Is there a mechanism to GC workflows that are completed?

Tables getting too big hasn't been a concern in practice because information on completed workflows can easily be GC'ed.

Re: Durable execution should be lightweight

#40

Post by WallWextra »

peterkelly wrote: post #8

Durable execution is best done at the level of a language implementation, not as a library. A workflow engine I recently built provided an interpreter for a Scheme-based language that, for each blocking operation, took a snapshot of the interpreter state (heap + stack) and persisted that to a database. Each time an operation completes (which could be after hours/days/weeks), the interpreter state is restored from the…

Racket's web framework has a feature to serialize continuations and restore them, which I instantly thought of the first time I saw durable execution.
Post reply on HN