Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building durable workflows on Postgres

dbos.dev

Post reply on HN
Print view View original

31–40 of 159 posts

Re: Building durable workflows on Postgres

#31

Post by PunchyHamster »

munk-a wrote: post #28
PunchyHamster wrote: post #25

Earlier quoted context omitted.

what's "maintenance" here ? If app is also using PostgreSQL it should be just initial effort of writing/importing code to run it, no ?

You pay for everything you build - the more complexity you put into it the more that costs over time. Dependencies need to be updated, language/framework upgrades usually break something, new features/requirements introduce additional complexity and code to manage. Software just costs money every day - not a lot, our industry is much lower margin than, say, stamping sheets of metal into tools - but it still has opera…

I know that. This looks like some lib you update once a year/every new CVE, and it is compared to a lib from cloud vendor and also update once a year/every new CVE, which is why I asked what it costed YOU in this particular case.

Re: Building durable workflows on Postgres

#32

Post by pphysch »

buremba wrote: post #27

All you need is Postgres until you scale into TBs of data. We use Postgresql as a durable workflow engine, vector search, time-series data, BM25 search, OLTP/OLAP engine, and a queue. It's basically the only dependency we have for https://lobu.ai The main benefit is centralizing all the data in one place so we don't need to worry about copying data in between multiple systems. Once something becomes the bottleneck, y…

I don't see logs mentioned. I agree with most those applications but would keep my OLAP stuff (metrics, logs, traces) in a separate store like VictoriaMetrics, both for capacity and read activity.

Re: Building durable workflows on Postgres

#33

Post by throwaway7783 »

buremba wrote: post #27

All you need is Postgres until you scale into TBs of data. We use Postgresql as a durable workflow engine, vector search, time-series data, BM25 search, OLTP/OLAP engine, and a queue. It's basically the only dependency we have for https://lobu.ai The main benefit is centralizing all the data in one place so we don't need to worry about copying data in between multiple systems. Once something becomes the bottleneck, y…

I'm in the same camp. Do you use any specific extensions? Especially for OLAP and time series (partitioned tables + related extensions work fine, but curious if you use anything else)

Re: Building durable workflows on Postgres

#34

Post by dakiol »

temporal_thr123 wrote: post #16
throwaw12 wrote: post #4

Curious to know experience of people using DBOS and Temporal. I have used Temporal in the past, works really good, my only problem with it was some limits on request payload or event sizes, created some inconveniences to us when building solutions. It also enforces good engineering practices, but sometimes you don't want to write special logic if your CSV file is larger than 2Mb, upload it to S3, pass link, then down…

I run a large on-prem temporal setup - throwaway acct as they will likely out me. Temporal is, in my opinion having run it in prod for over a year - poorly designed, slow and ridicliously heavy infra wise. If you're doing anything non-trivial (say, 200+ events/workflow) and you need to run only a couple hundred of them concurrently all day, you're going to spend millions on infra, and it's still going to absolutely s…

Agree. Have worked in a codebase using Temporal, and is pretty much a nightmare. I don't know about the infra side, but from the developer side, all the abstractions they bring to the table are poorly designed. Wouldn't recommend

Re: Building durable workflows on Postgres

#35

Post by pants2 »

throwaw12 wrote: post #4

Curious to know experience of people using DBOS and Temporal. I have used Temporal in the past, works really good, my only problem with it was some limits on request payload or event sizes, created some inconveniences to us when building solutions. It also enforces good engineering practices, but sometimes you don't want to write special logic if your CSV file is larger than 2Mb, upload it to S3, pass link, then down…

I thought Temporal was overly complex, but as you said the best part is it does enforce good engineering practices.

Then I tried their Cloud offering and was appalled at their pricing. I burned through the $1,000 free credits before I even got something to production. Didn't want to bother with running a local Temporal, either.

Best solution is to just take inspiration from their architecture and then do it yourself in Postgres, IMO.

Re: Building durable workflows on Postgres

#37

Post by stuartaxelowen »

hmaxdml wrote: post #30
buremba wrote: post #27

All you need is Postgres until you scale into TBs of data. We use Postgresql as a durable workflow engine, vector search, time-series data, BM25 search, OLTP/OLAP engine, and a queue. It's basically the only dependency we have for https://lobu.ai The main benefit is centralizing all the data in one place so we don't need to worry about copying data in between multiple systems. Once something becomes the bottleneck, y…

Listen/notify is poised to become much better in PG 18 and 19

Why’s that?

Re: Building durable workflows on Postgres

#39

Post by buremba »

pphysch wrote: post #32
buremba wrote: post #27

All you need is Postgres until you scale into TBs of data. We use Postgresql as a durable workflow engine, vector search, time-series data, BM25 search, OLTP/OLAP engine, and a queue. It's basically the only dependency we have for https://lobu.ai The main benefit is centralizing all the data in one place so we don't need to worry about copying data in between multiple systems. Once something becomes the bottleneck, y…

I don't see logs mentioned. I agree with most those applications but would keep my OLAP stuff (metrics, logs, traces) in a separate store like VictoriaMetrics, both for capacity and read activity.

Yeah I have logs in Sentry, which also uses Postgresql.

Re: Building durable workflows on Postgres

#40

Post by stuartaxelowen »

My dream is, instead of separating data storage, state machines, valid state constraints, and the logic that transitions between valid states, we can actually unify these into some kernel of app state. Honestly, Postgres already has a lot of these capabilities, but I don’t see an obvious story on the app or product level, providing provably correct sets of states that apps can transition between, and which they can automatically expose to clients in informative ways (this user can like this post, but not edit). It looks colored Petri net shaped to me, but I don’t yet see a simple app state paradigm in the same way that the database has obvious successful boundaries.
Post reply on HN