Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Build durable workflows with Postgres

dbos.dev

Post reply on HN
Print view View original

41–50 of 58 posts

Re: Build durable workflows with Postgres

#41

Post by sbrother »

mmcclure wrote: post #31
cpursley wrote: post #2

I've been using https://www.pgflow.dev for workflows which is built on pgmq and am really impressed so far. Most of the logic is in the database so I'm considering building an Elixir adapter DSL.

Just curious, if you’re already in Elixir and using Postgres, why not use Oban[1]? It’s my absolute favorite background job library, and the thing I often miss most when working in other ecosystems. [1] https://github.com/oban-bg/oban

Oban is so good! My startup has an extensive graph of background jobs all managed by Oban, and it's just rock solid, simple to use and gets out of the way.

Re: Build durable workflows with Postgres

#42

Post by digdugdirk »

atombender wrote: post #27
at0mic22 wrote: post #6

Every few years someone discovers FOR UPDATE SKIP LOCKED and represents it. I remember it lasting for 15 years at least

The "someone" in this case happens to be Michael Stonebraker, the creator of Postgres and CTO of DBOS.

So glad someone else chuckled reading this. Two thumbs up for knowing better than the creator of the thing they're talking about!

Re: Build durable workflows with Postgres

#43

Post by jedberg »

atombender wrote: post #28

While DBOS looks like a nice system, I was really disappointed to learn that Conductor, which is the DBOS equivalent of the Temporal server, is not open source. Without it, you get no centralized coordination of workflow recovery. On Kubernetes, for example, my understanding is that you will need to use a stateful set to assign stable executor IDs, which the Conductor doesn't need. I suppose that's their business mod…

> Conductor, which is the DBOS equivalent of the Temporal server,

Just to clarify, Conductor is not anything like the Temporal server. In Temporal, the server is a critical component that stores all the running state and is required for Temporal to work (and blocks your app from working if it's down).

Conductor is an out of band connector to give Transact users access to the same observability and workflow management as DBOS Cloud users have, but it isn't required and your app will keep working even if it breaks.

You can run a durable, and scalable, application with just Transact, it's just a lot harder without Conductor to help you.

You are correct that the business model is to provide add ons for Transact applications, but I'd say it's unfair to call Transact a "simplistic foundation" and not "grown up".

Transact is absolutely Enterprise grade software that can run at massive scale.

Re: Build durable workflows with Postgres

#44

Post by cyberpunk »

jumploops wrote: post #29

I've been looking at migrating to Temporal, but this looks interesting. For context, we have a simple (read: home-built) "durable" worker setup that uses BullMQ for scheduling/queueing, but all of the actual jobs are Postgres-based. Due to the cron-nature of the many disparate jobs (bespoke AI-native workflows), we have workers that scale up/down basically on the hour, every hour. Temporal is the obvious solution, bu…

Unless you’re planning on using their (temporalio’s) saas you’re in for building a very large database cluster for this if you need some scale.

(source: i run way more cassandra than i ever thought reasonable)

Re: Build durable workflows with Postgres

#45

Post by rubenvanwyk »

alpb wrote: post #8

I've been following DBOS for a while and I think the model isn't too different than Azure Durable Functions (which uses Azure Queues/Tables under the covers to maintain state). https://learn.microsoft.com/en-us/azure/azure-functions/dura... Perhaps the only difference is that Azure Durable Functions has more syntactic sugar in C# (instead of DBOS choice being Python) to preserve call results in the persistent storage…

This actually looks super amazing for C# ~ but doesn’t use Postgres?? All the backends seem to be purely Azure related / Microsoft products, so although the Framework is Apache2, your infrastructure needs to rely on MS?

Re: Build durable workflows with Postgres

#47

Post by qianli_cs »

rubenvanwyk wrote: post #46

Been looking at DBOS for a while ~ are there plans to port to other languages such as Java or C#?? Are you open to community ports??

Yeah, we plan to add more languages. Currently supports Python and TypeScript, and Go and Java will be released soon. We’re having a preview of DBOS Java at our user group meeting on August 28: https://lu.ma/8rqv5o5z Welcome to join us! We’d love to hear your feedback.

We welcome community contributions to the open source repos.

Re: Build durable workflows with Postgres

#48

Post by mnahkies »

jumploops wrote: post #29

I've been looking at migrating to Temporal, but this looks interesting. For context, we have a simple (read: home-built) "durable" worker setup that uses BullMQ for scheduling/queueing, but all of the actual jobs are Postgres-based. Due to the cron-nature of the many disparate jobs (bespoke AI-native workflows), we have workers that scale up/down basically on the hour, every hour. Temporal is the obvious solution, bu…

As another commentator said, temporal is quite tricky to self host/scale in a cost effective manner. This is also reflected in their cloud pricing (which should've been the warning sign to us tbh)

Overall it's a pretty heavy/expensive solution and I've come to the conclusion it's usage is best limited to lower frequency and/or higher "value" (eg: revenue or risk) tasks.

Orchestrating a food delivery that's paying you $3 of service fees - good use case. Orchestrating some high frequency task that pays you $3 / month - not so good.

Re: Build durable workflows with Postgres

#49

Post by cmdtab »

johtso wrote: post #34
cmdtab wrote: post #16

Recently moved some of the background jobs from graphile worker to DBOS. Really recommend for the simplicity. Took me half an hour. I evaluated temporal, trigger, cloudflare workflows (highly not recommended), etc and this was the easiest to implement incrementally. Didn't need to change our infrastructure at all. Just plugged the worker where I had graphile worker. The hosted service UX and frontend can use a lot of…

Why would you not recommend Cloudflare workflows? Was thinking of using them in my current project..

They inherit all the limitations of DO. For example, if you want to do anything that requires more than 6 TCP connection. Every fetch request will start failing silently because there is no more TCP connection to go through. This was a deal breaker for us. Their solution was split our code into more workflows or DOs.

You are limited to 128 MB ram which means everything has to be steamed. You will rewrite your code around this because many node libraries don't have streaming alternatives for things that spike memory usage.

The observability tab was buggy. Lifecycle chart is hard to understand for knowing when things will evict. Lot of small hidden limitations. Rate limits are very low for any mid scale application. Full Node compatibility is not there yet (work in progress) so needed to change some modules.

Overall, a gigantic waste of time unless you are doing something small scale. Just go with restate/upstash + lambdas/cloud run if you want simpler experience that scales in serverless manner

Re: Build durable workflows with Postgres

#50

Post by hanikesn »

KraftyOne wrote: post #36
krashidov wrote: post #35

How does this compare with inngest or restate? We currently use inngest right now and it works great but the typescript API is a bit clunky

Like Inngest and Restate, DBOS provides durable workflows. The difference is that DBOS is implemented as a Postgres-backed library you can "npm install" into your project (no external dependencies except Postgres), while Inngest and Restate require an external workflow orchestrator. Here's a blog post explaining the DBOS architecture in more detail: https://www.dbos.dev/blog/what-is-lightweight-durable-execut... Here…

First: It's great to have some serious competition in the durable execution market! You build some impressive tech already! It took me a decent time to investigate before pulling the trigger on temporal at my current PSP fintech as you'll end up with a pretty hard vendor lock-in which has to be worth it.

I have to say the architecture blog post is very leightweight on details, which makes it hard to judge so you have to dive into the docs for the spicy details and I see abig showstoppers for our usecase:

1. Workflow versioning: It seems currently impossible to properly upgrade inflight workflow without manually forking the workflow

Also the marketing material and docs aretoo handwavy on the idempotency and determinism constraints, which makes them dbos seem to good to be true. I'd also love to see higher level abstractions around message sending. Temporal's signal, query and update are essential building blocks for our use cases and allow some quite interesting use cases as you can really view workflows as actors.

I hope you can keep innovating as temporal really is too heavy for a lot of use-cases.

See: https://docs.dbos.dev/production/self-hosting/workflow-recov... https://www.dbos.dev/blog/handling-failures-workflow-forks

Post reply on HN