Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building durable workflows on Postgres

dbos.dev

Post reply on HN
Print view View original

91–100 of 159 posts

Re: Building durable workflows on Postgres

#91

Post by temporal_thr321 »

lll-o-lll wrote: post #57
temporal_thr123 wrote: post #16

Earlier quoted context omitted.

I run a large on-prem temporal setup - throwaway acct as they will likely out me. Temporal is, in my opinion having run it in prod for over a year - poorly designed, slow and ridicliously heavy infra wise. If you're doing anything non-trivial (say, 200+ events/workflow) and you need to run only a couple hundred of them concurrently all day, you're going to spend millions on infra, and it's still going to absolutely s…

> If you're doing anything non-trivial (say, 200+ events/workflow) and you need to run only a couple hundred of them concurrently all day, you're going to spend millions on infra, and it's still going to absolutely suck. Where are the “millions” on infra going? It’s a handful of services and a Postgres? > Their sales team is also absolutely appalling and desperate. You said “on-prem”. It’s open source; why are you de…

Not a couple hundred in one day, a couple hundred being started, concurrently, every second in a day. Each with ~200 events.

We need a 12 node cassandra cluster for this, with 64cpu nodes. So no, it's not a couple of services and a postgres.

Sales team, as we are an enterprise, and they want to extract money from us.

Re: Building durable workflows on Postgres

#92

Post by teaearlgraycold »

alexwennerberg wrote: post #86
tibbon wrote: post #52

Earlier quoted context omitted.

But when you hit that wall, it is hard to stop and convince people to use different patterns and systems. I've seen so many tables go from "it will only be a few thousand rows" to suddenly several TB and then people are looking confused when performance and db admin tasks get really difficult. I'm working at a scale where almost every day I have to ask people "are you use you need to treat that as relational data? It…

> But when you hit that wall, it is hard to stop and convince people to use different patterns and systems. I've seen so many tables go from "it will only be a few thousand rows" to suddenly several TB and then people are looking confused when performance and db admin tasks get really difficult. It's much, much worse in my experience to have to develop for the opposite -- working on a system that was designed for an…

[dead]

Re: Building durable workflows on Postgres

#93

Post by nicoburns »

cultofmetatron wrote: post #64
buremba wrote: post #27

All you need is Postgres until you scale into TBs of data. We use Postgresql as a durable workflow engine, vector search, time-series data, BM25 search, OLTP/OLAP engine, and a queue. It's basically the only dependency we have for https://lobu.ai The main benefit is centralizing all the data in one place so we don't need to worry about copying data in between multiple systems. Once something becomes the bottleneck, y…

conversely, startups that start scaling for tbs of data never make it to needing tbs of data. They burn too much energy scaling when they don't yet have a product people want yet.

Yep. I've also seen systems that were slow with <10GB data of because of bad application of patterns that were supposedly "scalable" (pulling entire tables out of the database to implement joins in application code because "nosql is faster" is not actually fast).

Re: Building durable workflows on Postgres

#94

Post by cyberpunk »

turtlebits wrote: post #69
lll-o-lll wrote: post #57

Earlier quoted context omitted.

> If you're doing anything non-trivial (say, 200+ events/workflow) and you need to run only a couple hundred of them concurrently all day, you're going to spend millions on infra, and it's still going to absolutely suck. Where are the “millions” on infra going? It’s a handful of services and a Postgres? > Their sales team is also absolutely appalling and desperate. You said “on-prem”. It’s open source; why are you de…

The same with any "open-source" enterprise ($$$) software. It sucks to run yourself. Docs on running/errors are non-existent. Their helm charts are broken. Instead of degraded performance, it just fails.

Yeah, they've had so much VC cash pumped in lately they really need to pump the SAAS side of the business.

Re: Building durable workflows on Postgres

#95

Post by hedora »

I want to dig into this "free" workflow_error.sql. I'll assume 1024 byte workflow job descriptors, and the article's steady state of 10,000 jobs per second.

Possibility one: There is one index on the table, and it is the created_at TS. This query has to scan 10,000 jobs/sec * 60 seconds * 60 minutes * 24 hours * 31 days * 1024 bytes / job = 25,543 GB.

A KV store would scan exactly that much.

Possibility two: The primary key is refined to (state, timestamp). Assume a 1% failure rate. Now, we "only" scan and return 255 GB. A key value store would scan exactly that much. (This is probably the right physical design).

Possibility three: The primary key is (timestamp), and there's a secondary index on state. I guess we do an index join, where one side of the join is 25,543 GB, and the other side is one unsorted bucket with 255GB * number of months the system has been in operation in it.

A KV store wouldn't let you express that.

Now, what other ad hoc queries are we supposed to efficiently support over a one month lookback? Also, what does PG do if you tell it to scan 25TB at the same time as it's inserting 10MB/sec at 10K TPS? How is vacuuming configured?

Re: Building durable workflows on Postgres

#96

Post by icedchai »

alexwennerberg wrote: post #86
tibbon wrote: post #52

Earlier quoted context omitted.

But when you hit that wall, it is hard to stop and convince people to use different patterns and systems. I've seen so many tables go from "it will only be a few thousand rows" to suddenly several TB and then people are looking confused when performance and db admin tasks get really difficult. I'm working at a scale where almost every day I have to ask people "are you use you need to treat that as relational data? It…

> But when you hit that wall, it is hard to stop and convince people to use different patterns and systems. I've seen so many tables go from "it will only be a few thousand rows" to suddenly several TB and then people are looking confused when performance and db admin tasks get really difficult. It's much, much worse in my experience to have to develop for the opposite -- working on a system that was designed for an…

[dead]

Re: Building durable workflows on Postgres

#97

Post by agumonkey »

tibbar wrote: post #77
agumonkey wrote: post #65

Earlier quoted context omitted.

was it due to the language expressiveness forcing too much verbosity ? (honest question)

lack of version control, clunky language mechanics, performance issues, etc.

hmm lack of version may be tooled (reminds me of smalltalk, lisp file-less culture)

Re: Building durable workflows on Postgres

#98

Post by hmaxdml »

joshka wrote: post #10

This feels like the sort of architecture that starts clean and then gradually grows most of the things a workflow-native system already has. I've seen systems like this, seen companies that are built out of this idea, and built small systems like this over time. Once you need retries, backoff, timeouts, cancellation, versioning, visibility, task routing, rate limits, leases, heartbeats, stuck-worker detection, replay…

Yeah, we've observed that too: people start implementing their own retry logic, idempotency, etc. But then they grow a hard to maintain, complex stack that's not their core business logic. There's a reason why there is a dedicated team building DBOS, every day. Because it's not that easy to build a solid durable workflows engine on Postgres.

Re: Building durable workflows on Postgres

#99

Post by hmaxdml »

OutOfHere wrote: post #29

I am not convinced that using a special software for "durable workflows" is necessary. If one has a stateful message queue or job task queue, e.g. RabbitMQ or Celery, one can use it. Irrespective, many jobs can be made idempotent. The most that you ought to residually need is a column in an existing table of your own database which keeps track of what remains to be done. Given the above, it would seem that durable wo…

I've talked to dozens of engineers who built their home grown "durable" stack. Most of them eventually moved on to buying vs building, when their system actually scaled. It's just not a side-hustle to build a foundational reliability layer.

Re: Building durable workflows on Postgres

#100

Post by lll-o-lll »

temporal_thr321 wrote: post #91
lll-o-lll wrote: post #57

Earlier quoted context omitted.

> If you're doing anything non-trivial (say, 200+ events/workflow) and you need to run only a couple hundred of them concurrently all day, you're going to spend millions on infra, and it's still going to absolutely suck. Where are the “millions” on infra going? It’s a handful of services and a Postgres? > Their sales team is also absolutely appalling and desperate. You said “on-prem”. It’s open source; why are you de…

Not a couple hundred in one day, a couple hundred being started, concurrently, every second in a day. Each with ~200 events. We need a 12 node cassandra cluster for this, with 64cpu nodes. So no, it's not a couple of services and a postgres. Sales team, as we are an enterprise, and they want to extract money from us.

We’re all enterprise.

If you have 200 WF’s/sec each with 200 events, it sounds to me that you have a sizeable amount of work flowing through this system. 17 million workflows per day? Can I call these transactions?

Do these transactions add value to your business? Do you need durable execution for all these workloads?

Temporal is just a tool; and like any tool it can be misused. For the classic “book a hotel + airline, handle the partial failures” case, 17 million bookings a day would imply you should be thrilled with Temporal.

If you are using it to perform WAF in a firewall; you would be less thrilled. The scale you are describing, and that you aren’t super excited about the incredible amount of money pouring in, makes me question if the use-cases are fitting the tool.

Post reply on HN