Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Build durable workflows with Postgres

dbos.dev

Post reply on HN
Print view View original

21–30 of 58 posts

Re: Build durable workflows with Postgres

#21

Post by cmdtab »

LudwigNagasena wrote: post #19
cmdtab wrote: post #16

Recently moved some of the background jobs from graphile worker to DBOS. Really recommend for the simplicity. Took me half an hour. I evaluated temporal, trigger, cloudflare workflows (highly not recommended), etc and this was the easiest to implement incrementally. Didn't need to change our infrastructure at all. Just plugged the worker where I had graphile worker. The hosted service UX and frontend can use a lot of…

What was the reason for the transition?

Needed checkpoints in some of our jobs wrapping around the AI agent so we can reduce cost and increase reliability (as workflow will start from mid step as opposed to a complete restart).

We already check pointed the agent but then figure it's better to have a generic abstraction for other stuff we do.

Re: Build durable workflows with Postgres

#22

Post by diarrhea »

cmdtab wrote: post #16

Recently moved some of the background jobs from graphile worker to DBOS. Really recommend for the simplicity. Took me half an hour. I evaluated temporal, trigger, cloudflare workflows (highly not recommended), etc and this was the easiest to implement incrementally. Didn't need to change our infrastructure at all. Just plugged the worker where I had graphile worker. The hosted service UX and frontend can use a lot of…

Interesting!

What made you opt for DBOS over Temporal?

Re: Build durable workflows with Postgres

#23

Post by cmdtab »

diarrhea wrote: post #22
cmdtab wrote: post #16

Recently moved some of the background jobs from graphile worker to DBOS. Really recommend for the simplicity. Took me half an hour. I evaluated temporal, trigger, cloudflare workflows (highly not recommended), etc and this was the easiest to implement incrementally. Didn't need to change our infrastructure at all. Just plugged the worker where I had graphile worker. The hosted service UX and frontend can use a lot of…

Interesting! What made you opt for DBOS over Temporal?

Temporal required re-architecting some stuff, their typescript sdk and sandbox is bit unintuitive to use so would have been an additional item to grok for the team, and additional infrastructure to maintain. There was a latency trade off too which in our case mattered.

Didn't face any issue though. Temporal observability and UI was better than DBOS. Just harder to do incremental migration in an existing codebase.

Re: Build durable workflows with Postgres

#24

Post by darkteflon »

Often wondered whether it would be possible / advisable to combine DBOS with, e.g., Dagster if you have complex data orchestration requirements. They seem to deal with orthogonal concerns but complement nicely. Is integration with orchestration frameworks something the DBOS team has any thoughts on?

Re: Build durable workflows with Postgres

#25

Post by KraftyOne »

darkteflon wrote: post #24

Often wondered whether it would be possible / advisable to combine DBOS with, e.g., Dagster if you have complex data orchestration requirements. They seem to deal with orthogonal concerns but complement nicely. Is integration with orchestration frameworks something the DBOS team has any thoughts on?

Would love to learn more about what you're building--what problems or parts of your system would you solve with Dagster vs DBOS?

Re: Build durable workflows with Postgres

#28

Post by atombender »

While DBOS looks like a nice system, I was really disappointed to learn that Conductor, which is the DBOS equivalent of the Temporal server, is not open source.

Without it, you get no centralized coordination of workflow recovery. On Kubernetes, for example, my understanding is that you will need to use a stateful set to assign stable executor IDs, which the Conductor doesn't need.

I suppose that's their business model, to provide a simplistic foundation where you have to pay money to get the grown up stuff.

Re: Build durable workflows with Postgres

#29

Post by jumploops »

I've been looking at migrating to Temporal, but this looks interesting.

For context, we have a simple (read: home-built) "durable" worker setup that uses BullMQ for scheduling/queueing, but all of the actual jobs are Postgres-based.

Due to the cron-nature of the many disparate jobs (bespoke AI-native workflows), we have workers that scale up/down basically on the hour, every hour.

Temporal is the obvious solution, but it will take some rearchitecting to get our jobs to fit their structure. We're also concerned with some of their limits (payload size, language restrictions, etc.).

Looking at DBOS, it's unclear from the docs how to scale the workers:

> DBOS is just a library for your program to import, so it can run with any Python/Node program.

In our ideal case, we can add DBOS to our main application for scheduling jobs, and then have a simple worker app that scales independently.

How "easy" would it be to migrate our current system to DBOS?

Re: Build durable workflows with Postgres

#30

Post by KraftyOne »

jumploops wrote: post #29

I've been looking at migrating to Temporal, but this looks interesting. For context, we have a simple (read: home-built) "durable" worker setup that uses BullMQ for scheduling/queueing, but all of the actual jobs are Postgres-based. Due to the cron-nature of the many disparate jobs (bespoke AI-native workflows), we have workers that scale up/down basically on the hour, every hour. Temporal is the obvious solution, bu…

I'd love to learn more about what you're building--just reach out at peter.kraft@dbos.dev.

One option is that you have DBOS workflows that schedule and submit jobs to an external worker app. Another option is that your workers use DBOS queues (https://docs.dbos.dev/python/tutorials/queue-tutorial). I'd have to better understand your use case to figure out what would be the best fit.

Post reply on HN