Having used several external orchestrators I can see the appeal of the simplicity of this approach, especially for smaller teams wanting to limit the amount of infrastructure to maintain. Postgres is a proven tool, and as long as you design `@step`s to each perform one non deterministic side effect, I can see this scaling very well both in terms of performance and maintainability.
Durable execution should be lightweight
11–20 of 45 posts
Re: Durable execution should be lightweight
#12Durable execution is best done at the level of a language implementation, not as a library. A workflow engine I recently built provided an interpreter for a Scheme-based language that, for each blocking operation, took a snapshot of the interpreter state (heap + stack) and persisted that to a database. Each time an operation completes (which could be after hours/days/weeks), the interpreter state is restored from the…
I see it being a trade-off between how explicit the state persisted for a workflow execution is (rows in a database for Temporal and DBOS) vs how natural it is to write such a workflow (like in your PL/compiler). Given workflows are primarily used for business use-cases, with a lot of non-determinacy coming from interaction with third-party services or other deployments, the library implementation feels more appropriate.
Though I am assuming building durability at a language-level means the whole program state must be serializable, which sounds tricky. Curious if you could share more?
Re: Durable execution should be lightweight
#13My immediate reaction is hell no. > In some sense, external orchestration turns individual applications into distributed microservices, with all the complexity that implies. I'd argue that durable execution intrisically is complex and external orchestrators give you tools to manage that complexity, whereas this attempts to brush the complexity under the rug in a way that does not inspire confidence.
What value does an external orchestrator add for managing complexity that an “embedded” solution could not?
The embedded seems to ignore that most modern systems are distributed now and an operation will spawn multiple services. So an external system is better for that usecase I would say.
Re: Durable execution should be lightweight
#14Durable execution is best done at the level of a language implementation, not as a library. A workflow engine I recently built provided an interpreter for a Scheme-based language that, for each blocking operation, took a snapshot of the interpreter state (heap + stack) and persisted that to a database. Each time an operation completes (which could be after hours/days/weeks), the interpreter state is restored from the…
I see it being a trade-off between how explicit the state persisted for a workflow execution is (rows in a database for Temporal and DBOS) vs how natural it is to write such a workflow (like in your PL/compiler). Given workflows are primarily used for business use-cases, with a lot of non-determinacy coming from interaction with third-party services or other deployments, the library implementation feels more appropri…
There's certainly a tradeoff between the two approaches; a simpler representation (list of tasks or DAG) is easier to query and manipulate, at the cost of being less expressive, lacking features like loops, conditionals, etc.
In the workflow engine I described, state is represented as a graph of objects in memory; this includes values like integers/strings and data structures like dictionaries/lists, as well as closures, environments, and the execution stack. This graph is serialised as JSON and stored in a postgres table. A more compact binary representation could be added in the future if performance requirements demand it, but JSON has been sufficient for our needs so far. A delta between each snapshot is also stored in an execution log, so that the complete execution history is stored for auditing purposes.
The interpreter is written in such a way that all object allocation, object manipulation, and garbage collection is under its control, and all the data needed to represent execution state is stored in a manner that can be easily serialised. In particular, we avoid the use of pointers to memory locations, instead using object ids for all references. So the persistent state, when loaded, can be accessed directly, since any time a reference from one object to another needs to be followed, the interpreter does so by looking up the object in the heap based on its id.
Non-deterministic and blocking operations (including IPC receives) are handled outside of the evaluation cycle. This enables their results to be explicitly captured in the execution log, and allows for retries to be handled by an external mechanism under control of the user (since retrying can be unsafe if the operation is not idempotent).
The biggest win of using a proper language for expressing the workflow is the ability to add arbitrary logic between blocking operations, such as conditional tests or data structure manipulation. Any kind of logic you might want to do can be expressed due to the fact the workflow language is Turing-complete.
Re: Durable execution should be lightweight
#15Hot take: this is bad architecture.
The solution seems to be solving for the simplest use case (internal stateless functions) rather than the most complex use case (external state-impactful functions).
Furthermore, the words used aren't really what they should be talking about.
>> Because workflows are just Python functions, the thread can restart a workflow by simply calling the workflow function with its original inputs and ID, retrieved from Postgres.
>> For this model to work, we have to make one assumption: workflow functions must be deterministic.
Yes, "deterministic"... because all state modification is aligned to ensure that.
If instead of a single print() the function/step had 2 print()'s, the state leaks and the abstraction explodes.
The right abstraction here is probably something more functional / rust-like, where external state modifications sections are explicitly decorated (either automatically or by a developer).
Re: Durable execution should be lightweight
#16Earlier quoted context omitted.
I see it being a trade-off between how explicit the state persisted for a workflow execution is (rows in a database for Temporal and DBOS) vs how natural it is to write such a workflow (like in your PL/compiler). Given workflows are primarily used for business use-cases, with a lot of non-determinacy coming from interaction with third-party services or other deployments, the library implementation feels more appropri…
There's certainly a tradeoff between the two approaches; a simpler representation (list of tasks or DAG) is easier to query and manipulate, at the cost of being less expressive, lacking features like loops, conditionals, etc. In the workflow engine I described, state is represented as a graph of objects in memory; this includes values like integers/strings and data structures like dictionaries/lists, as well as closu…
That's really interesting! It does seem that this is identically semantically to the library approach (as the logic your interpreter adds around steps could also be added by decorators) but is completely automatic. Which is great if the interpreter always does the right thing, but problematic/overly magical if the interpreter doesn't. For example, if your problem domain has two blocking operations that really form one single step and should be retried together, a library approach lets you express that but an interpreted approach might get it wrong.
Re: Durable execution should be lightweight
#17Hot take: this is bad architecture. The solution seems to be solving for the simplest use case (internal stateless functions) rather than the most complex use case (external state-impactful functions). Furthermore, the words used aren't really what they should be talking about. >> Because workflows are just Python functions, the thread can restart a workflow by simply calling the workflow function with its original i…
That's exactly what this model is! The @Step decorator is for external state modifications. Then @Workflows orchestrate steps. The example shows the simplest possible external state modification--a print to the terminal.
Steps can be tried multiple times (if a failure happens mid-step) but never re-execute once complete. Since idempotency can't be added externally, that's the strongest possible guarantee any orchestration system can give you (and if your step is performing an idempotent operation, which is the safest thing, you can use the workflow ID as an idempotency key). More details in the docs: https://docs.dbos.dev/python/tutorials/workflow-tutorial#rel...
Re: Durable execution should be lightweight
#18I think the example given in this blog post might need a "health warning" that steps should, generally, be doing more than just printing "hello". I can imagine that the reads and writes to Postgres for a large number of workflows, each with a large number of small steps called in a tight loop, would cause some significant performance problems. The examples given on their main site are a little more meaningful.
Yes, that's totally fair. Usually, a step is a meaningful unit of work, such as a API call that performs an external state modification. Because each step is a fair chunk of work, and the overhead is just one write per step, this scales well in practice--as well as Postgres scales, up to 10K+ operations/second.
Re: Durable execution should be lightweight
#19This seems like temporal only without as much server and complexity. Maybe they ignore it or it really is that simple. Overall really cool! There are some scalability concerns that are brought that I think are valid but maybe you have a Postgres server backing up every few servers that need this kind of execution. Also, every function shouldn't be its own step but needs to be divided into larger chunks where every re…
Thanks! DBOS is simpler not because it ignores complexity, but because it uses Postgres to deal with complexity. And Postgres is a very powerful tool for building reliable systems!
Re: Durable execution should be lightweight
#20This seems like temporal only without as much server and complexity. Maybe they ignore it or it really is that simple. Overall really cool! There are some scalability concerns that are brought that I think are valid but maybe you have a Postgres server backing up every few servers that need this kind of execution. Also, every function shouldn't be its own step but needs to be divided into larger chunks where every re…
The example is overly simplified. It glosses over many of the subtle-but-important aspects of durable execution. For example: - Steps should be small but fallible operations - eg. sending a request to an external service. You generally want to tailor the retry logic on steps to the specific task they are doing. Doing too much in a step can increase failure rates or cause other problems due to the at-least-once behavi…
This is a great answer, and yes, those are critical aspects of durable execution. Maybe I should write a follow-on post that goes into more detail...