Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dbos: Durable Workflow Orchestration with Go and PostgreSQL

github.com

Post reply on HN
Print view View original

31–40 of 62 posts

Re: Dbos: Durable Workflow Orchestration with Go and PostgreSQL

#31

Post by qianli_cs »

plmpsu wrote: post #29
hmaxdml wrote: post #2

Thanks for posting! I am one of the author, happy to answer any question!

How does DBOS scale in a cluster? with Temporal or Dapr Workflows, applications register running their supported workflows types or activities and the workflow orchestration framework balances work across applications. How does this work in the library approach? Also, how is DBOS handling workflow versioning? Looking forward for your Java implementation. Thanks

Good questions!

DBOS naturally scales to distributed environments, with many processes/servers per application and many applications running together. The key idea is to use the database concurrency control to coordinate multiple processes. [1]

When a DBOS workflow starts, it’s tagged with the version of the application process that launched it. This way, you can safely change workflow code without breaking existing ones. They'll continue running on the older version. As a result, rolling updates become easy and safe. [2]

[1] https://docs.dbos.dev/architecture#using-dbos-in-a-distribut...

[2] https://docs.dbos.dev/architecture#application-and-workflow-...

Re: Dbos: Durable Workflow Orchestration with Go and PostgreSQL

#32

Post by plmpsu »

qianli_cs wrote: post #31
plmpsu wrote: post #29

Earlier quoted context omitted.

How does DBOS scale in a cluster? with Temporal or Dapr Workflows, applications register running their supported workflows types or activities and the workflow orchestration framework balances work across applications. How does this work in the library approach? Also, how is DBOS handling workflow versioning? Looking forward for your Java implementation. Thanks

Good questions! DBOS naturally scales to distributed environments, with many processes/servers per application and many applications running together. The key idea is to use the database concurrency control to coordinate multiple processes. [1] When a DBOS workflow starts, it’s tagged with the version of the application process that launched it. This way, you can safely change workflow code without breaking existing…

Thanks for the reply.

So applications continuously poll the database for work? Have you done any benchmarking to evaluate the throughput of DBOS when running many workflows, activities, etc.?

Re: Dbos: Durable Workflow Orchestration with Go and PostgreSQL

#36

Post by qianli_cs »

plmpsu wrote: post #32
qianli_cs wrote: post #31

Earlier quoted context omitted.

Good questions! DBOS naturally scales to distributed environments, with many processes/servers per application and many applications running together. The key idea is to use the database concurrency control to coordinate multiple processes. [1] When a DBOS workflow starts, it’s tagged with the version of the application process that launched it. This way, you can safely change workflow code without breaking existing…

Thanks for the reply. So applications continuously poll the database for work? Have you done any benchmarking to evaluate the throughput of DBOS when running many workflows, activities, etc.?

In DBOS, workflows can be invoked directly as normal function calls or enqueued. Direct calls don't require any polling. For queued workflows, each process runs a lightweight polling thread that checks for new work using `SELECT ... FOR UPDATE SKIP LOCKED` with exponential backoffs to prevent contentions, so many concurrent workers can poll efficiently. We recently wrote a blog post on durable workflows, queues, and optimizations: https://www.dbos.dev/blog/why-postgres-durable-execution

Throughput mainly comes down to database writes: executing a workflow = 2 writes (input + output), each step = 1 write. A single Postgres instance can typically handle thousands of writes per second, and a larger one can handle tens of thousands (or even more, depending on your workload size). If you need more capacity, you can shard your app across multiple Postgres servers.

Re: Dbos: Durable Workflow Orchestration with Go and PostgreSQL

#38

Post by hazn »

hmaxdml wrote: post #2

Thanks for posting! I am one of the author, happy to answer any question!

I remember reading that restate.dev is a 'push' based workflow and therefore works well with serverless workflows: https://news.ycombinator.com/item?id=40660568

what is your input on these two topics? aka pull vs push and working well with serverless workflows

Re: Dbos: Durable Workflow Orchestration with Go and PostgreSQL

#39

Post by intermerda »

hmaxdml wrote: post #2

Thanks for posting! I am one of the author, happy to answer any question!

I remembered reading about the DBOS paper a while back - https://arxiv.org/abs/2007.11112. Is this an evolution of that research work? If so, how did an OS for databases morph into a workflow orchestration service?

Re: Dbos: Durable Workflow Orchestration with Go and PostgreSQL

#40

Post by bjornsing »

qianli_cs wrote: post #30
bjornsing wrote: post #27

Earlier quoted context omitted.

> Yes, in any durability framework there's still the possibility that a process crashes mid-step, in which case you have no choice but to restart the step. Golem [1] is an interesting counterexample to this. They run your code in a WASM runtime and essentially checkpoint execution state at every interaction with the outside world. But it seems they are having trouble selling into the workflow orchestration market. Pe…

I think one potential concern with "checkpoint execution state at every interaction with the outside world" is the size of the checkpoints. Allowing users to control the granularity by explicitly specifying the scope of each step seems like a more flexible model. For example, you can group multiple external interactions into a single step and only checkpoint the final result, avoiding the overhead of saving intermedi…

Sure you get more control with explicit state management. But it’s also more work, and more difficult work. You can do a lot of writes to NVMe for one developer salary.
Post reply on HN