Build durable workflows with Postgres
1–10 of 58 posts
Re: Build durable workflows with Postgres
#2I've been using https://www.pgflow.dev for workflows which is built on pgmq and am really impressed so far. Most of the logic is in the database so I'm considering building an Elixir adapter DSL.
Re: Build durable workflows with Postgres
#3I've been using https://www.pgflow.dev for workflows which is built on pgmq and am really impressed so far. Most of the logic is in the database so I'm considering building an Elixir adapter DSL.
what are you using the DSL for?
Re: Build durable workflows with Postgres
#4Anything that guarantees exactly once is selling snake oil. Side effects happen inside any transaction, and only when it commits (checkpoints) are the side effects safe.
Want to send an email, but the app crashes before committing? Now you're at-least-once.
You can compress the window that causes at-least-once semantics, but it's always there. For this reason, this blog post oversells the capabilities of these types of systems as a whole. DBOS (and Inngest, see the disclaimer below) try to get as close to exactly once as possible, but the risk always exists, which is why you should always try to use idempotency in external API requests if they support it. Defense in layers.
Disclaimer: I built the original `step.run` APIs at https://www.inngest.com, which offers similar things on any platform... without being tied to DB transactions.
Re: Build durable workflows with Postgres
#5Why not just use Temporal?
Re: Build durable workflows with Postgres
#6Every few years someone discovers FOR UPDATE SKIP LOCKED and represents it. I remember it lasting for 15 years at least
Re: Build durable workflows with Postgres
#7Anything that guarantees exactly once is selling snake oil. Side effects happen inside any transaction, and only when it commits (checkpoints) are the side effects safe. Want to send an email, but the app crashes before committing? Now you're at-least-once. You can compress the window that causes at-least-once semantics, but it's always there. For this reason, this blog post oversells the capabilities of these types…
As the post says, the exactly-once guarantee is ONLY for steps performing database operations. For those, you actually can get an exactly-once guarantee by running the database operations in the same Postgres transaction as your durable checkpoint. That's a pretty cool benefit of building workflows on Postgres! Of course, if there are side effects outside the database, those happen at-least-once.
Re: Build durable workflows with Postgres
#8I've been following DBOS for a while and I think the model isn't too different than Azure Durable Functions (which uses Azure Queues/Tables under the covers to maintain state).
https://learn.microsoft.com/en-us/azure/azure-functions/dura...
Perhaps the only difference is that Azure Durable Functions has more syntactic sugar in C# (instead of DBOS choice being Python) to preserve call results in the persistent storage? Where else do they differ? At the end, all of them seem to be doing what Temporal is doing (which has its own shortcomings and it's also possible to get it wrong if you call a function directly instead of invoking it via an Activity etc)?
Re: Build durable workflows with Postgres
#9Anything that guarantees exactly once is selling snake oil. Side effects happen inside any transaction, and only when it commits (checkpoints) are the side effects safe. Want to send an email, but the app crashes before committing? Now you're at-least-once. You can compress the window that causes at-least-once semantics, but it's always there. For this reason, this blog post oversells the capabilities of these types…
As the post says, the exactly-once guarantee is ONLY for steps performing database operations. For those, you actually can get an exactly-once guarantee by running the database operations in the same Postgres transaction as your durable checkpoint. That's a pretty cool benefit of building workflows on Postgres! Of course, if there are side effects outside the database, those happen at-least-once.
You can totally leverage postgres transactions to give someone... postgres transactions!
I just figured that the exactly once semantics were so worth discussing that any external side effects (which is what orchestration is for) aren't included in that, which is a big caveat.
Re: Build durable workflows with Postgres
#10Anything that guarantees exactly once is selling snake oil. Side effects happen inside any transaction, and only when it commits (checkpoints) are the side effects safe. Want to send an email, but the app crashes before committing? Now you're at-least-once. You can compress the window that causes at-least-once semantics, but it's always there. For this reason, this blog post oversells the capabilities of these types…
[deleted]