Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building durable workflows on Postgres

dbos.dev

Post reply on HN
Print view View original

131–140 of 159 posts

Re: Building durable workflows on Postgres

#131

Post by potamic »

temporal_thr123 wrote: post #16
throwaw12 wrote: post #4

Curious to know experience of people using DBOS and Temporal. I have used Temporal in the past, works really good, my only problem with it was some limits on request payload or event sizes, created some inconveniences to us when building solutions. It also enforces good engineering practices, but sometimes you don't want to write special logic if your CSV file is larger than 2Mb, upload it to S3, pass link, then down…

I run a large on-prem temporal setup - throwaway acct as they will likely out me. Temporal is, in my opinion having run it in prod for over a year - poorly designed, slow and ridicliously heavy infra wise. If you're doing anything non-trivial (say, 200+ events/workflow) and you need to run only a couple hundred of them concurrently all day, you're going to spend millions on infra, and it's still going to absolutely s…

> if the sales team call you make sure legal is in the room.

What's the deal? It couldn't harm just listening to sales, could it?

I presume legal would it be involved before anything is signed in any case?

Re: Building durable workflows on Postgres

#133

Post by sgt »

bootsmann wrote: post #44
sgt wrote: post #2

Continuously amazed by what you can do with few tools, as long as Postgres is a part of your toolkit. I recently developed a distributed queue and it works really great - benchmarks great too, with no race conditions or conflicts. I used SKIP LOCKED so that workers can compete safely. You can also have multiple workers across nodes avoid conflict by using session wide mutexes i.e. pg advisory lock.

Advisory locks are preferred for this anyways because holding a lot of SELECT FOR UPDATE doesn’t scale too well. Edit: Actually I checked this again and apparently the advice has now changed to the inverse.

I need to do proper benchmarks on SELECT FOR UPDATE..SKIP LOCKED - but I suspect thousands per second. Some claimed higher than 10k/sec, but we'll see

Re: Building durable workflows on Postgres

#134

Post by aryehof »

My fear is that durable workflows are increasingly being seen as required for everything, because we need to solve the distributed transaction problem in a micro-services world.

It questions the initial wisdom of creating lots of little independent distributed apps, without regards to interaction between them. Let’s build ever more necessary plumbing and schemes just to enable their interaction.

I am arguing that durable workflows should be a last resort for boundaries you must cross, not a default pattern for every business process.

Re: Building durable workflows on Postgres

#136

Post by quacker »

temporal_thr321 wrote: post #88
quacker wrote: post #83

Earlier quoted context omitted.

Honest question: Can you use Temporal Cloud? Have you evaluated Temporal Cloud pricing? Ballparking: 200 events/workflow, 200 workflows/per day and assuming 1 event = 1 cloud action[1], that is 1.2M or so actions per month. The $100/month plan includes 1M actions each month, and even the pay-as-you pricing when you exceed that is $50 per 1M actions[2]. Temporal Cloud seems extremely cheap for your use case, even if I…

I was not clear; I did not mean not 200 a day, it's 10s of thousands of concurrently running workflows, sometimes into the hundreds of thousands, each with 200 events. We run many hundreds of thousands of these a day. Temporal was a bad fit for us, and we regret it deeply.

Ah. So multiple billion actions per month, and probably multiple million dollars per year on their cloud, if they can even support that load (plus, the vendor lock in and etc). Makes sense.

Re: Building durable workflows on Postgres

#137

Post by cpursley »

joshka wrote: post #10

This feels like the sort of architecture that starts clean and then gradually grows most of the things a workflow-native system already has. I've seen systems like this, seen companies that are built out of this idea, and built small systems like this over time. Once you need retries, backoff, timeouts, cancellation, versioning, visibility, task routing, rate limits, leases, heartbeats, stuck-worker detection, replay…

https://github.com/pgmq/pgmq

Re: Building durable workflows on Postgres

#138

Post by eecc »

hilariously wrote: post #13
switchbak wrote: post #7

Earlier quoted context omitted.

They've just released an external storage approach to solve the large payload issue. I don't 100% love it (it's bolted on, not an intrinsic part), and it's an early release right now - but you can consider this effectively solved for now.

That's good because back in the day if you were putting entire documents in a message queue I would laugh people out the door, putting something in object storage + linking is much more useful (though the distributed system part/backup current state part can be annoying!)

It’s an EIP, called Claim Check.

https://www.enterpriseintegrationpatterns.com/patterns/messa...

Re: Building durable workflows on Postgres

#139

Post by cpursley »

Thaxll wrote: post #116

All those solutions based on PG are missing the point, you need a good SDK so that devs can create those workflow without re-inventing the wheel: error, retries, observability, idempotency ect ... PG is just a detail of implementation, you need a good library to build reliable flows.

https://github.com/agoodway/pgflow#2-define-a-flow

https://www.pgflow.dev/get-started/flows/create-flow/#the-gr...

Re: Building durable workflows on Postgres

#140

Post by tomcam »

joshka wrote: post #10

This feels like the sort of architecture that starts clean and then gradually grows most of the things a workflow-native system already has. I've seen systems like this, seen companies that are built out of this idea, and built small systems like this over time. Once you need retries, backoff, timeouts, cancellation, versioning, visibility, task routing, rate limits, leases, heartbeats, stuck-worker detection, replay…

Ridiculously good analysis! HN is a national treasure because of posts like this.
Post reply on HN