Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building durable workflows on Postgres

dbos.dev

Post reply on HN
Print view View original

41–50 of 159 posts

Re: Building durable workflows on Postgres

#41

Post by bcooke »

sorentwo wrote: post #8
senderista wrote: post #3

Citing CockroachDB as an example of scaling Postgres made me spit out coffee. Was this LLM-written?

The efforts we've undergone to make Oban (and Pro) work with CRDB have been ridiculous. Feature detection all over because of a lack of common operators and functions that can't be used in indexes. The worst is the rampant "serialization_failure" errors that force continual transaction retries. Not how I'd suggest scaling Postgres. That said, as a predecessor to dbos in building durable workflows just using Postgres,…

Can you expand on why you chose to use CRDB with Oban? I have no opinion here, I’m genuinely curious as someone using Oban myself (with Postgres). I haven’t hit the point of really needing to scale it out yet and I’d rather avoid the traps others have figured out.

Re: Building durable workflows on Postgres

#43

Post by buremba »

throwaway7783 wrote: post #33
buremba wrote: post #27

All you need is Postgres until you scale into TBs of data. We use Postgresql as a durable workflow engine, vector search, time-series data, BM25 search, OLTP/OLAP engine, and a queue. It's basically the only dependency we have for https://lobu.ai The main benefit is centralizing all the data in one place so we don't need to worry about copying data in between multiple systems. Once something becomes the bottleneck, y…

I'm in the same camp. Do you use any specific extensions? Especially for OLAP and time series (partitioned tables + related extensions work fine, but curious if you use anything else)

The native extensions are fine but I don't have good experience with any third party extensions, so far tried Timescale, pg_lake, citus, and pgvectorscale. They look very appealing but it's usually a trap as you can't get the value without using the vendor's cloud offerings.

I think if you grow enough to look for these extensions, it's usually better to bet on purpose-specific tooling. For example, I use DuckDB/Iceberg combination extensively for columnar data and connect DuckDB to PG when I need it.

Re: Building durable workflows on Postgres

#44

Post by bootsmann »

sgt wrote: post #2

Continuously amazed by what you can do with few tools, as long as Postgres is a part of your toolkit. I recently developed a distributed queue and it works really great - benchmarks great too, with no race conditions or conflicts. I used SKIP LOCKED so that workers can compete safely. You can also have multiple workers across nodes avoid conflict by using session wide mutexes i.e. pg advisory lock.

Advisory locks are preferred for this anyways because holding a lot of SELECT FOR UPDATE doesn’t scale too well.

Edit: Actually I checked this again and apparently the advice has now changed to the inverse.

Re: Building durable workflows on Postgres

#45

Post by TkTech »

stuartaxelowen wrote: post #37
hmaxdml wrote: post #30

Earlier quoted context omitted.

Listen/notify is poised to become much better in PG 18 and 19

Why’s that?

In pg19 https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit... will land, which significantly improves NOTIFY performance. Right now LISTEN/NOTIFY doesn't scale to very busy instances because a `NOTIFY` within a transaction takes a global lock.

Re: Building durable workflows on Postgres

#46

Post by ivanr »

TkTech wrote: post #45
stuartaxelowen wrote: post #37

Earlier quoted context omitted.

Why’s that?

In pg19 https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit... will land, which significantly improves NOTIFY performance. Right now LISTEN/NOTIFY doesn't scale to very busy instances because a `NOTIFY` within a transaction takes a global lock .

More context: https://www.recall.ai/blog/postgres-listen-notify-does-not-s...

Re: Building durable workflows on Postgres

#47

Post by mrits »

hbarka wrote: post #12

How do you incorporate secrets in this kind of implementation? Stored in db?

Unless you have a very specific use case, you wouldn't want to store in db or in any message you use in any workflow like this. Usually whatever does the actual work has a way to get the secret.

Re: Building durable workflows on Postgres

#49

Post by halamadrid »

We work on disk log based architecture for workflows at Unmeshed (https://unmeshed.io/) which helps it to scale at a fraction of the cost of traditional workflow systems that are based on expensive databases.

Postgres is not cheap to run in the cloud at scale. We went for the cheapest infra, which is basically the disk storage.

Re: Building durable workflows on Postgres

#50

Post by pragma_x »

I completely get the concept and agree - this is great way to build this kind of durability in a workflow system.

That said, my gamer-brain wants to call this "Save-scumming at scale." Which is to say, a lot of people already know that this approach works, but maybe they haven't made the connection to abstract CS stuff.

Another strategy that can be used to build robustness is to build your workflow out of idempotent operations. That can be useful for situations where the workflow state is too large to back up. Instead, you just run the job from the top and it's a bunch of no-ops until you start making progress again.

Post reply on HN