Building a modern durable execution engine from first principles

Can you elaborate more on your persistence layer?

One of the good reasons why most products will layer on top of an established database like Postgres is because concerns like ACID are solved problems in those databases, and the database itself is well battle-hardened, Jepsen-tested with a reputation for reliability, etc. One of the reasons why many new database startups fail is precisely because it is so difficult to get over that hump with potential customers - you can't really improve reliability until you run into production bugs, and you can't sell because it's not reliable. It's a tough chicken-and-egg problem.

I appreciate you have reasons to build your own persistence layer here (like a push-based model), but doesn't doing so expose you to the same kind of risk as a new database startup? Particularly when we're talking about a database for durable execution, for which, you know, durability is a hard requirement?

afaik, with Temporal you deploy workers. When a workflow calls an activity, the activity gets added to a queue, and the workers pull activities from queues.

In Restate, there are no workers like that. The durable functions (which contain the equivalent of the activity logic) get deployed on FaaS or like a containerized RPC service. The Restate broker calls the function/service with the argument and some attached context (journal, state, ...).

You can think of it a bit like Kafka vs. EventBridge. The former needs long lived clients that poll for events, the latter pushes events to subscribers/listeners.

This "push" (Restate broker calls the service) means there doesn't have to be a long running process waiting for work (by polling a queue).

I think the difference also naturally from the programming abstraction: In Temporal, it is workflows that create activities, in Restate it stateful durable functions (bundled into services).

I just realized I missed an important part: The primary durability for the bulk of the state comes from S3 (or similar object store). The periodic snapshots give you like an automatic frequent backup mechanism for free, which in itself is a nice property to have.

I have to say, the examples are really good: https://github.com/restatedev/examples

For someone (me) who is new to distributed systems, and what durable execution even is, I learned a lot (both from the blog post & the examples!). thanks!

ChatGPT also helped :)

(DBOS co-founder here) DBOS embeds durable execution into your app as a library backed by Postgres, whereas Restate provides durable execution as a service.

In my opinion, this makes DBOS more lightweight and easier to integrate into an existing application. To use DBOS, you just install the library and annotate workflows and steps in your program. DBOS will checkpoint your workflows and steps in Postgres to make them durable and recover them from failures, but will otherwise leave your application alone. By contrast, to use Restate, you need to split out your durable code into a separate worker service and use the Restate server to dispatch events to it. You're essentially outsourcing control flow and event processing to the Restate server., which will require some rearchitecting.

Here's a blog post with more detail comparing DBOS and Temporal, whose model is similar to (but not the same as!) Restate: https://www.dbos.dev/blog/durable-execution-coding-compariso...