Seems like an interesting idea to add durability and resumability to lengthy cron jobs.
pg_durable: Microsoft open sources in-database durable execution
21–30 of 119 posts
Re: pg_durable: Microsoft open sources in-database durable execution
#22I'm trapped on Azure at work and we're constantly waiting for Azure pg to catch up with modernity.
For example, you cant use this: https://www.paradedb.com/blog/hybrid-search-in-postgresql-th...
Also for example, you dont get ultra-wide high dimensionality vectors.
It is nice they are open sourcing pg_durable, but how about adopting table stakes I'd get with AWS?
Re: pg_durable: Microsoft open sources in-database durable execution
#232026 is the year of the Postgres queue! (DBOS[0], pgQue[1]) It's awesome that the community is contributing this and giving us the option to use it. As an ex-app engineer though, I kind of prefer my queue logic to be in code, in Git, but maybe with the right tooling, you can change my mind. :) [0]: https://www.dbos.dev/ [1]: https://github.com/NikolayS/pgque
Yeah, it's harder to work on, or maybe just different, but I guess the docs, info(searchable docs, posts, experience), and tooling are lacking.
What's the story for version control, debugging, testing, releasing? It'd be cool to have everything together for data locality and simplifying the stack, but it feels you'd lose a lot of useful knowledge about how to do stuff "properly".
Re: pg_durable: Microsoft open sources in-database durable execution
#24Isn't the database already one of the hardest piece of infras to scale? Why would you want to load it with additional long-running jobs?
Re: pg_durable: Microsoft open sources in-database durable execution
#25This feels like the wrong solution to an age old problem solved by the DAG schedulers like Apache Airflow for a while now. Why would I want to store my control flow in the database and not in code? It feels strange. Not trying to dismiss the project, I'm just not getting it yet I think.
Microsoft has their own Durable Task framewor[1] for that kind of stuff, and it supports both running as a self-hosted standalone service like temporal, and running serverless on Azure Functions. It actually predated airflow, temporal, etc., IIRC.
This one seems to be more database-specific use case. The advantage is probably that you can track the exact state of the job in the database itself, rather than having to cross-reference the workflow log with the codebase and trace through it line by line to figure out what the state is. Plus I assume it's less overhead and latency, and operationally one less thing to spin up.
[1] https://learn.microsoft.com/en-us/azure/durable-task/common/...
Re: pg_durable: Microsoft open sources in-database durable execution
#26Isn't the database already one of the hardest piece of infras to scale? Why would you want to load it with additional long-running jobs?
Gotta set up the fall to rake in the dough later with consulting fees
Re: pg_durable: Microsoft open sources in-database durable execution
#27Looks pretty good but I wonder why they didn’t build it on pgmq? If you’re on elixir I maintain a DAG package around this (based on and compatible with pgflow.dev which is TS/Deno). https://github.com/agoodway/pgflow
(pg_durable committer here) The provider is an extensibility point. We just shipped the simplest version of it. Happy to take contribs if someone sends a pgmq based provider!
When do we get mssql_durable :)
Re: pg_durable: Microsoft open sources in-database durable execution
#282026 is the year of the Postgres queue! (DBOS[0], pgQue[1]) It's awesome that the community is contributing this and giving us the option to use it. As an ex-app engineer though, I kind of prefer my queue logic to be in code, in Git, but maybe with the right tooling, you can change my mind. :) [0]: https://www.dbos.dev/ [1]: https://github.com/NikolayS/pgque
+1 on "prefer my queue logic to be in code". The of my data doesn't change nearly as much as the actions I need to take on it; it doesn't make sense to me why I'd want to do a migration (which is an all or nothing op btw) every time I want to change how I behave with my data. This is also why I absolutely abhorred having to make postgres functions to do anything remotely non-trivial on Supabase.
That said, we did hand-build a simple job queue (just lock, poll, reserve on a column, poll and update reservation to mark job done) on top of postgres at my previous startup. Something like pgque would have made that much more polished.
Re: pg_durable: Microsoft open sources in-database durable execution
#292026 is the year of the Postgres queue! (DBOS[0], pgQue[1]) It's awesome that the community is contributing this and giving us the option to use it. As an ex-app engineer though, I kind of prefer my queue logic to be in code, in Git, but maybe with the right tooling, you can change my mind. :) [0]: https://www.dbos.dev/ [1]: https://github.com/NikolayS/pgque
Yeah, it's harder to work on, or maybe just different, but I guess the docs, info(searchable docs, posts, experience), and tooling are lacking. What's the story for version control, debugging, testing, releasing? It'd be cool to have everything together for data locality and simplifying the stack, but it feels you'd lose a lot of useful knowledge about how to do stuff "properly".
Contributor here. Good points, we do need to develop some best practices around managing function versioning and lifecycle for pg_durable.
https://github.com/microsoft/duroxide - also OSS, the durable execution framework pg_durable is built on itself supports function versions. We can leverage that to get similar support in pg_durable.
Re: pg_durable: Microsoft open sources in-database durable execution
#30Can anyone explain why I would want to use this over an orchestration tool that lives outside the DB? Read through the Readme and some of the examples, I still don't get it.
Snapshot PITR of your database means everything restores including the durable jobs at the PIT. Don't need to synchronize the backups with anything else that is part of the same data store, good for ETL pipelines and other state machine type jobs. If your ETL is mostly SQL anyway, then having the actual job being run on the same server helps as well.
Yes, but that doesn't have to imply that the compute part of the durable jobs framework also needs to be part of the database snapshot. You almost certainly want that defined in code anyway, if only to have a sane versioning story. So then by having it also be part of the snapshot, you've now got the problem that there are apparently two sources of truth for that bit of the code.