Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building durable workflows on Postgres

dbos.dev

Post reply on HN
Print view View original

51–60 of 159 posts

Re: Building durable workflows on Postgres

#51

Post by vrm »

llimllib wrote: post #26

Armin Ronacher's `absurd` is an implementation of durable workflows for postgres: https://lucumr.pocoo.org/2025/11/3/absurd-workflows/ https://github.com/earendil-works/absurd https://earendil-works.github.io/absurd/ I've not used it, but it's worth comparing to other options

If you don't need a ton of throughput I think `absurd` (and our Rust derivative `durable`) are very nice options that keep the client side extremely simple. It's also lightweight enough that a coding agent can keep the entire thing in its head easily and just run queries to look up state as needed.

Re: Building durable workflows on Postgres

#52

Post by tibbon »

buremba wrote: post #27

All you need is Postgres until you scale into TBs of data. We use Postgresql as a durable workflow engine, vector search, time-series data, BM25 search, OLTP/OLAP engine, and a queue. It's basically the only dependency we have for https://lobu.ai The main benefit is centralizing all the data in one place so we don't need to worry about copying data in between multiple systems. Once something becomes the bottleneck, y…

But when you hit that wall, it is hard to stop and convince people to use different patterns and systems. I've seen so many tables go from "it will only be a few thousand rows" to suddenly several TB and then people are looking confused when performance and db admin tasks get really difficult.

I'm working at a scale where almost every day I have to ask people "are you use you need to treat that as relational data? It doesn't seem relational"

Re: Building durable workflows on Postgres

#53

Post by TkTech »

pphysch wrote: post #32
buremba wrote: post #27

All you need is Postgres until you scale into TBs of data. We use Postgresql as a durable workflow engine, vector search, time-series data, BM25 search, OLTP/OLAP engine, and a queue. It's basically the only dependency we have for https://lobu.ai The main benefit is centralizing all the data in one place so we don't need to worry about copying data in between multiple systems. Once something becomes the bottleneck, y…

I don't see logs mentioned. I agree with most those applications but would keep my OLAP stuff (metrics, logs, traces) in a separate store like VictoriaMetrics, both for capacity and read activity.

pg_timescale can take you pretty far for metrics and would be Good Enough for almost all users. Totally agree on raw, high-volume logs though.

Re: Building durable workflows on Postgres

#54

Post by TkTech »

bcooke wrote: post #41
sorentwo wrote: post #8

Earlier quoted context omitted.

The efforts we've undergone to make Oban (and Pro) work with CRDB have been ridiculous. Feature detection all over because of a lack of common operators and functions that can't be used in indexes. The worst is the rampant "serialization_failure" errors that force continual transaction retries. Not how I'd suggest scaling Postgres. That said, as a predecessor to dbos in building durable workflows just using Postgres,…

Can you expand on why you chose to use CRDB with Oban? I have no opinion here, I’m genuinely curious as someone using Oban myself (with Postgres). I haven’t hit the point of really needing to scale it out yet and I’d rather avoid the traps others have figured out.

sorentwo is the author of Oban. He's not using CockroachDB, he's supporting it as a valid Oban target.

Re: Building durable workflows on Postgres

#55

Post by Linell »

temporal_thr123 wrote: post #16
throwaw12 wrote: post #4

Curious to know experience of people using DBOS and Temporal. I have used Temporal in the past, works really good, my only problem with it was some limits on request payload or event sizes, created some inconveniences to us when building solutions. It also enforces good engineering practices, but sometimes you don't want to write special logic if your CSV file is larger than 2Mb, upload it to S3, pass link, then down…

I run a large on-prem temporal setup - throwaway acct as they will likely out me. Temporal is, in my opinion having run it in prod for over a year - poorly designed, slow and ridicliously heavy infra wise. If you're doing anything non-trivial (say, 200+ events/workflow) and you need to run only a couple hundred of them concurrently all day, you're going to spend millions on infra, and it's still going to absolutely s…

[dead]

Re: Building durable workflows on Postgres

#56

Post by tibbar »

stuartaxelowen wrote: post #40

My dream is, instead of separating data storage, state machines, valid state constraints, and the logic that transitions between valid states, we can actually unify these into some kernel of app state. Honestly, Postgres already has a lot of these capabilities, but I don’t see an obvious story on the app or product level, providing provably correct sets of states that apps can transition between, and which they can a…

This has been tried, but thousand-line stored procedures are truly a nightmare.

Re: Building durable workflows on Postgres

#57

Post by lll-o-lll »

temporal_thr123 wrote: post #16
throwaw12 wrote: post #4

Curious to know experience of people using DBOS and Temporal. I have used Temporal in the past, works really good, my only problem with it was some limits on request payload or event sizes, created some inconveniences to us when building solutions. It also enforces good engineering practices, but sometimes you don't want to write special logic if your CSV file is larger than 2Mb, upload it to S3, pass link, then down…

I run a large on-prem temporal setup - throwaway acct as they will likely out me. Temporal is, in my opinion having run it in prod for over a year - poorly designed, slow and ridicliously heavy infra wise. If you're doing anything non-trivial (say, 200+ events/workflow) and you need to run only a couple hundred of them concurrently all day, you're going to spend millions on infra, and it's still going to absolutely s…

> If you're doing anything non-trivial (say, 200+ events/workflow) and you need to run only a couple hundred of them concurrently all day, you're going to spend millions on infra, and it's still going to absolutely suck.

Where are the “millions” on infra going? It’s a handful of services and a Postgres?

> Their sales team is also absolutely appalling and desperate.

You said “on-prem”. It’s open source; why are you dealing with their sales team?

> If you're doing anything non-trivial (say, 200+ events/workflow) and you need to run only a couple hundred of them concurrently all day…

If “millions” were required to obtain such tiny scale, I’d agree there’d be a massive problem. No one would use Temporal; it would be a complete waste of resource. If this were true.

Re: Building durable workflows on Postgres

#58

Post by saxenaabhi »

As someone who uses dbos.dev, restate.dev, cf workflows here is a snippet from our Agents.md:

  Restate.dev:
    for payment integrations on northflank since its faster than cf workflows, independent of cf and its downtime and self-hostable vendor-lock-in free,
  Cloudflare workflows:
    for non critical stuff like csv/pdf report generations since it's very cheap.
  DBOS.dev:
    for workflows that need atomic messaging tied to a postgres db transaction for 100% reliabilty/durabilty(for example populating a materialized row or sending out critical email/push to a merchant).
DBOS and Restate are similar on surface but Restate requires a central "orchestrator" which has pros and cons but makes it easy to build with serverless workers on cf/vercel.

It also has VirtualObject which is a nice vendor-lock-in-free OSS alternative to CF's single threaded DurableObject.

Where DBOS absolutely shines is

1) Atomic messaging in the same db tx as your business logic via dbos.enqueue_workflow! This is often the most brittle part of any solution and doing it atomically and durably with same tx that ran your business logic drastically reduces lots of complexity.

2) Since DBOS stores workflow state in db it should be easy to build dashboard for observability from metabase/looker(I wish restate exposed its rocksdb instance so it could be hooked up to metabase).

Re: Building durable workflows on Postgres

#59

Post by opiniateddev »

opiniateddev wrote: post #17

Conductor OSS does this quite well https://docs.conductor-oss.org/devguide/ai/index.html https://github.com/agentspan-ai/agentspan which is essentially an agentic SDK layer for Conductor can convert any of your langgraph, openAI, vercel, or ADK agent and makes it durable and adds orchestration with no code changes.

for our production we use Redis for queues but have seen users using both Postgres and MySQL for queues as well.

Re: Building durable workflows on Postgres

#60

Post by cpursley »

rafael-lua wrote: post #48

The "everything can be done in Postgres" crowd is crazy. It is like a religion at this point.

But that's not what we are saying; we're suggesting use Postgres until you truly need something else. 90% of applications aren't "web scale", keep the stack simple and portable. There's no good reason to slap in a ton of moving parts until they are truly needed.
Post reply on HN