Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building durable workflows on Postgres

dbos.dev

Post reply on HN
Print view View original

81–90 of 159 posts

Re: Building durable workflows on Postgres

#81

Post by turkeyboi »

dieselgate wrote: post #63
tibbon wrote: post #52

Earlier quoted context omitted.

But when you hit that wall, it is hard to stop and convince people to use different patterns and systems. I've seen so many tables go from "it will only be a few thousand rows" to suddenly several TB and then people are looking confused when performance and db admin tasks get really difficult. I'm working at a scale where almost every day I have to ask people "are you use you need to treat that as relational data? It…

> are you use you need to treat that as relational data? Is this intended to be "you sure you need..."?

Obviously, yes

Re: Building durable workflows on Postgres

#82

Post by bob1029 »

tibbar wrote: post #77
agumonkey wrote: post #65

Earlier quoted context omitted.

was it due to the language expressiveness forcing too much verbosity ? (honest question)

lack of version control, clunky language mechanics, performance issues, etc.

Version control might not be a big deal if you are all-in on the database.

Stored procedures are easiest to version by simply defining multiple variants and then incrementally moving the callers in the direction you want. The durability comes from (hopefully) your backups. Point-in-time-recovery is often easier for the business to reason about than a git repository.

Re: Building durable workflows on Postgres

#83

Post by quacker »

temporal_thr123 wrote: post #16
throwaw12 wrote: post #4

Curious to know experience of people using DBOS and Temporal. I have used Temporal in the past, works really good, my only problem with it was some limits on request payload or event sizes, created some inconveniences to us when building solutions. It also enforces good engineering practices, but sometimes you don't want to write special logic if your CSV file is larger than 2Mb, upload it to S3, pass link, then down…

I run a large on-prem temporal setup - throwaway acct as they will likely out me. Temporal is, in my opinion having run it in prod for over a year - poorly designed, slow and ridicliously heavy infra wise. If you're doing anything non-trivial (say, 200+ events/workflow) and you need to run only a couple hundred of them concurrently all day, you're going to spend millions on infra, and it's still going to absolutely s…

Honest question: Can you use Temporal Cloud? Have you evaluated Temporal Cloud pricing?

Ballparking: 200 events/workflow, 200 workflows/per day and assuming 1 event = 1 cloud action[1], that is 1.2M or so actions per month. The $100/month plan includes 1M actions each month, and even the pay-as-you pricing when you exceed that is $50 per 1M actions[2].

Temporal Cloud seems extremely cheap for your use case, even if I'm off by a factor of 10. Is there a catch? You still need infra to run your Temporal workers, and I assume there are storage and other costs, but I assume action usage is the majority of it.

1. Not sure exactly what constitutes an "Action". At a glance, seems like most events have a corresponding action(?) and a subset of those actions are actually billable(?)

2. https://docs.temporal.io/cloud/pricing#payg-action-pricing

Re: Building durable workflows on Postgres

#84

Post by gjvc »

sroussey wrote: post #62
buremba wrote: post #27

All you need is Postgres until you scale into TBs of data. We use Postgresql as a durable workflow engine, vector search, time-series data, BM25 search, OLTP/OLAP engine, and a queue. It's basically the only dependency we have for https://lobu.ai The main benefit is centralizing all the data in one place so we don't need to worry about copying data in between multiple systems. Once something becomes the bottleneck, y…

Use different “databases” besides public at the very start. No joins between them. You will be in a good position to just split the postgres instance by those at a later date. They will have different usage patterns than the merged version you have now, and will be easier to optimize and will buy you some time. And time is all you need.

"public" is not a database, it is a schema within a database.

apropos bad naming, postgresql authors are not forgiven for naming all the databases on a single host a "cluster". I mean __really__.

Re: Building durable workflows on Postgres

#85

Post by valyala »

buremba wrote: post #39
pphysch wrote: post #32

Earlier quoted context omitted.

I don't see logs mentioned. I agree with most those applications but would keep my OLAP stuff (metrics, logs, traces) in a separate store like VictoriaMetrics, both for capacity and read activity.

Yeah I have logs in Sentry, which also uses Postgresql.

Sentry stores logs in ClickHouse - https://blog.sentry.io/how-sentry-queries-unstructured-data-...

Re: Building durable workflows on Postgres

#86

Post by alexwennerberg »

tibbon wrote: post #52
buremba wrote: post #27

All you need is Postgres until you scale into TBs of data. We use Postgresql as a durable workflow engine, vector search, time-series data, BM25 search, OLTP/OLAP engine, and a queue. It's basically the only dependency we have for https://lobu.ai The main benefit is centralizing all the data in one place so we don't need to worry about copying data in between multiple systems. Once something becomes the bottleneck, y…

But when you hit that wall, it is hard to stop and convince people to use different patterns and systems. I've seen so many tables go from "it will only be a few thousand rows" to suddenly several TB and then people are looking confused when performance and db admin tasks get really difficult. I'm working at a scale where almost every day I have to ask people "are you use you need to treat that as relational data? It…

> But when you hit that wall, it is hard to stop and convince people to use different patterns and systems. I've seen so many tables go from "it will only be a few thousand rows" to suddenly several TB and then people are looking confused when performance and db admin tasks get really difficult.

It's much, much worse in my experience to have to develop for the opposite -- working on a system that was designed for an imagined "infinite" scale that in reality like 100GB and a few transactions a minute.

Re: Building durable workflows on Postgres

#87

Post by cyberpunk »

lll-o-lll wrote: post #57
temporal_thr123 wrote: post #16

Earlier quoted context omitted.

I run a large on-prem temporal setup - throwaway acct as they will likely out me. Temporal is, in my opinion having run it in prod for over a year - poorly designed, slow and ridicliously heavy infra wise. If you're doing anything non-trivial (say, 200+ events/workflow) and you need to run only a couple hundred of them concurrently all day, you're going to spend millions on infra, and it's still going to absolutely s…

> If you're doing anything non-trivial (say, 200+ events/workflow) and you need to run only a couple hundred of them concurrently all day, you're going to spend millions on infra, and it's still going to absolutely suck. Where are the “millions” on infra going? It’s a handful of services and a Postgres? > Their sales team is also absolutely appalling and desperate. You said “on-prem”. It’s open source; why are you de…

We also hit scaling problems with temporal.

Postgres doesn't scale at all four our workload, so you're into cassandra.

For a medium sized deployment, you're looking at 200+ vcpus, and then lets say standard dev/uat/prod. So now you're at 600 cpus. Now you need two geographic regions, dev can stay in one place, so now you're at 800. Want a failover cluster for prod? Have another 200 cpus.

and 200 CPUs is a medium deployment, assuming something like 36 cpus per cassandra node, then say 4-8 per instance of matching, worker, history, frontend. Then all your other components around it, ingress controller, service mesh, etc.

There's a million a year easy, for a small deployment.

Our prod one is 4x this size.

Re: Building durable workflows on Postgres

#88

Post by temporal_thr321 »

quacker wrote: post #83
temporal_thr123 wrote: post #16

Earlier quoted context omitted.

I run a large on-prem temporal setup - throwaway acct as they will likely out me. Temporal is, in my opinion having run it in prod for over a year - poorly designed, slow and ridicliously heavy infra wise. If you're doing anything non-trivial (say, 200+ events/workflow) and you need to run only a couple hundred of them concurrently all day, you're going to spend millions on infra, and it's still going to absolutely s…

Honest question: Can you use Temporal Cloud? Have you evaluated Temporal Cloud pricing? Ballparking: 200 events/workflow, 200 workflows/per day and assuming 1 event = 1 cloud action[1], that is 1.2M or so actions per month. The $100/month plan includes 1M actions each month, and even the pay-as-you pricing when you exceed that is $50 per 1M actions[2]. Temporal Cloud seems extremely cheap for your use case, even if I…

I was not clear; I did not mean not 200 a day, it's 10s of thousands of concurrently running workflows, sometimes into the hundreds of thousands, each with 200 events. We run many hundreds of thousands of these a day.

Temporal was a bad fit for us, and we regret it deeply.

Re: Building durable workflows on Postgres

#89

Post by devstein »

quard8 wrote: post #15
throwaw12 wrote: post #4

Curious to know experience of people using DBOS and Temporal. I have used Temporal in the past, works really good, my only problem with it was some limits on request payload or event sizes, created some inconveniences to us when building solutions. It also enforces good engineering practices, but sometimes you don't want to write special logic if your CSV file is larger than 2Mb, upload it to S3, pass link, then down…

we're using dbos for ai gen workflows and processing video files. understanding how to migrate from celery took time, but for our case it was worth it.

If anyone else is looking to migrate from Celery to DBOS, we also made this transition and wrote about it https://dosu.dev/blog/migrate-celery-to-dbos-dosu

Very happy we made the switch.

Re: Building durable workflows on Postgres

#90

Post by everforward »

iwwff wrote: post #71

Every time I am surprised to see the promises of the cheap durability without mentioning costs of running dirable postgres, which might be not easy or cheap.

Postgres is durable by default, it's ACID compliant. It is not reliable in the HA sense by default.

Either way, I'd bet a hosted Postgres with HA is cheaper than whatever PaaS you're thinking of.

Post reply on HN