Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building durable workflows on Postgres

dbos.dev

Post reply on HN
Print view View original

111–120 of 159 posts

Re: Building durable workflows on Postgres

#111

Post by nulltrace »

joshka wrote: post #10

This feels like the sort of architecture that starts clean and then gradually grows most of the things a workflow-native system already has. I've seen systems like this, seen companies that are built out of this idea, and built small systems like this over time. Once you need retries, backoff, timeouts, cancellation, versioning, visibility, task routing, rate limits, leases, heartbeats, stuck-worker detection, replay…

The SKIP LOCKED pattern is fine until the worker count climbs. Then vacuum can't keep up. Dead tuples pile up, visibility map turns to swiss cheese. Queue table is tiny on disk but the planner thinks it's huge and stops using the index. It gets ugly fast.

Re: Building durable workflows on Postgres

#112

Post by buremba »

valyala wrote: post #85
buremba wrote: post #39

Earlier quoted context omitted.

Yeah I have logs in Sentry, which also uses Postgresql.

Sentry stores logs in ClickHouse - https://blog.sentry.io/how-sentry-queries-unstructured-data-...

Thanks for correcting me, I was wondering why I got downvoted. :)

Clickhouse is also great for OLAP for sure

Re: Building durable workflows on Postgres

#113

Post by jiggunjer »

dakiol wrote: post #34
temporal_thr123 wrote: post #16

Earlier quoted context omitted.

I run a large on-prem temporal setup - throwaway acct as they will likely out me. Temporal is, in my opinion having run it in prod for over a year - poorly designed, slow and ridicliously heavy infra wise. If you're doing anything non-trivial (say, 200+ events/workflow) and you need to run only a couple hundred of them concurrently all day, you're going to spend millions on infra, and it's still going to absolutely s…

Agree. Have worked in a codebase using Temporal, and is pretty much a nightmare. I don't know about the infra side, but from the developer side, all the abstractions they bring to the table are poorly designed. Wouldn't recommend

Biggest design bug imo is the workers need to register for the workflows they support, but will happily pull tasks from unrelated workflows if they're on the same queue. No way to put failed tasks back into the queue again either.

Re: Building durable workflows on Postgres

#114

Post by bcooke »

TkTech wrote: post #54
bcooke wrote: post #41

Earlier quoted context omitted.

Can you expand on why you chose to use CRDB with Oban? I have no opinion here, I’m genuinely curious as someone using Oban myself (with Postgres). I haven’t hit the point of really needing to scale it out yet and I’d rather avoid the traps others have figured out.

sorentwo is the author of Oban. He's not using CockroachDB, he's supporting it as a valid Oban target.

Ah ok thanks for the clarification. And thank you sorentwo for your fantastic work – I've been loving my switch to the Elixir ecosystem thanks to the efforts of folks like you.

Re: Building durable workflows on Postgres

#115

Post by [deleted] »

cyberpunk wrote: post #87
lll-o-lll wrote: post #57

Earlier quoted context omitted.

> If you're doing anything non-trivial (say, 200+ events/workflow) and you need to run only a couple hundred of them concurrently all day, you're going to spend millions on infra, and it's still going to absolutely suck. Where are the “millions” on infra going? It’s a handful of services and a Postgres? > Their sales team is also absolutely appalling and desperate. You said “on-prem”. It’s open source; why are you de…

We also hit scaling problems with temporal. Postgres doesn't scale at all four our workload, so you're into cassandra. For a medium sized deployment, you're looking at 200+ vcpus, and then lets say standard dev/uat/prod. So now you're at 600 cpus. Now you need two geographic regions, dev can stay in one place, so now you're at 800. Want a failover cluster for prod? Have another 200 cpus. and 200 CPUs is a medium depl…

[deleted]

Re: Building durable workflows on Postgres

#116

Post by Thaxll »

All those solutions based on PG are missing the point, you need a good SDK so that devs can create those workflow without re-inventing the wheel: error, retries, observability, idempotency ect ...

PG is just a detail of implementation, you need a good library to build reliable flows.

Re: Building durable workflows on Postgres

#117

Post by jedberg »

Thaxll wrote: post #116

All those solutions based on PG are missing the point, you need a good SDK so that devs can create those workflow without re-inventing the wheel: error, retries, observability, idempotency ect ... PG is just a detail of implementation, you need a good library to build reliable flows.

DBOS is that library.

Re: Building durable workflows on Postgres

#119

Post by Rapzid »

sgt wrote: post #2

Continuously amazed by what you can do with few tools, as long as Postgres is a part of your toolkit. I recently developed a distributed queue and it works really great - benchmarks great too, with no race conditions or conflicts. I used SKIP LOCKED so that workers can compete safely. You can also have multiple workers across nodes avoid conflict by using session wide mutexes i.e. pg advisory lock.

Just do a reservation on the record with the actor ID.

Re: Building durable workflows on Postgres

#120

Post by halfcat »

throwaw12 wrote: post #4

Curious to know experience of people using DBOS and Temporal. I have used Temporal in the past, works really good, my only problem with it was some limits on request payload or event sizes, created some inconveniences to us when building solutions. It also enforces good engineering practices, but sometimes you don't want to write special logic if your CSV file is larger than 2Mb, upload it to S3, pass link, then down…

DBOS is much less complexity compared to Temporal. That’s the benefit.

Main tradeoff is lower performance. Or at least, you’re going to be limited to what you can push through Postgres. If that’s sufficient for your needs DBOS is great.

Post reply on HN