Is this an open sourcing of something they use internally? My first thought on durable jobs was GHA aka Azure Devops.
Please see https://learn.microsoft.com/en-us/azure/horizondb/ai/ai-pipe...
Thanks for answering, this makes tons of sense
pg_durable: Microsoft open sources in-database durable execution
41–50 of 119 posts
Re: pg_durable: Microsoft open sources in-database durable execution
#42Earlier quoted context omitted.
Microsoft has their own Durable Task framewor[1] for that kind of stuff, and it supports both running as a self-hosted standalone service like temporal, and running serverless on Azure Functions. It actually predated airflow, temporal, etc., IIRC. This one seems to be more database-specific use case. The advantage is probably that you can track the exact state of the job in the database itself, rather than having to…
(Author of both durable task framework and pg_durable/duroxide here) Indeed Durable tasks is an exceptional project and was a unique innovation at the time. pg_durable brings the same reliability and durablity semantics to long running operations within the database. We have tons of interesting scenarios on the roadmap. Stay tuned! :)
Does ai.backfill() fill incomplete/dirty rows or does pg_durable have some notion of partial completion?
Re: pg_durable: Microsoft open sources in-database durable execution
#43Can anyone explain why I would want to use this over an orchestration tool that lives outside the DB? Read through the Readme and some of the examples, I still don't get it.
Contributor here. At Microsoft, our Postgres customers seem to split pretty evenly into 2 camps, those that want to do as much as they can in the database, and those that agree with your take - want to keep apps and compute outside the DB.
I bet this is correlated with how much they like/know Postgres already. When people don’t understand their database’s features, they want it to behave like something else they do understand (code). They’re leaving a lot of performance on the table by not leveraging everything their database can do.
Re: pg_durable: Microsoft open sources in-database durable execution
#44I'm trapped on Azure at work and we're constantly waiting for Azure pg to catch up with modernity. For example, you cant use this: https://www.paradedb.com/blog/hybrid-search-in-postgresql-th... Also for example, you dont get ultra-wide high dimensionality vectors. It is nice they are open sourcing pg_durable, but how about adopting table stakes I'd get with AWS?
Wouldn't Azure Cosmos DB be better suited for vector searches?
Re: pg_durable: Microsoft open sources in-database durable execution
#45A few things are not clear to me from reading through docs and examples:
df.wait_for_schedule()
For this[0]:
-- Wait for human signal (5 minute timeout)
~> (df.wait_for_signal('approval', 300) |=> 'sig')
~> df.if(
$$SELECT NOT ($sig::jsonb->>'timed_out')::boolean
AND ($sig::jsonb->'data'->>'approved')::boolean$$,
Also not immediately clear: how to handle errors/exceptions?
[0] https://github.com/microsoft/pg_durable/blob/main/examples/i...
Re: pg_durable: Microsoft open sources in-database durable execution
#46A few things are not clear to me from reading through docs and examples: df.wait_for_schedule() How does this call work? Is it idempotent if I call it from an application? If I run it 2x with the same parameters, does it double tick? Am I invoking this manually from a query console to only do this one time? Am I running this as part of a migration script? For this[0]: -- Wait for human signal (5 minute timeout) ~> (d…
You are creating a durable function and starting its execution at the same time by calling df.start(). This will you give you back an instance id which represents this durable function execution. You can use this to refer to this execution from this point onwards.
Within this durable function you are calling df.wait_for_signal(). This call is exactly once within this function instance. There are no duplicates possible. Your df.start() call might get duplicated if it times out and you re-run it, but in this case it would end up creating a different function instance.
Any 'unhandled' errors in executing SQL will fail the function instance. Its status would bubble up the exact error being raised.
Re: pg_durable: Microsoft open sources in-database durable execution
#47Earlier quoted context omitted.
(Author of both durable task framework and pg_durable/duroxide here) Indeed Durable tasks is an exceptional project and was a unique innovation at the time. pg_durable brings the same reliability and durablity semantics to long running operations within the database. We have tons of interesting scenarios on the roadmap. Stay tuned! :)
Does ai.backfill() fill incomplete/dirty rows or does pg_durable have some notion of partial completion?
Hi there! PM from the PG AI team, working on both pg_durable and the AI pipeline layer.
ai.backfill() ignores that row-level state entirely and reprocesses everything from scratch. https://learn.microsoft.com/en-us/azure/horizondb/ai/ai-pipe...
pg_durable answers "did this workflow instance finish, and if it crashed, where do I resume?", completed/running/pending/failed per node + checkpoint replay. https://github.com/microsoft/pg_durable/blob/main/USER_GUIDE...
If you want this problem addressed better, please add an issue to the open-source repo, we would love to dig in. https://github.com/microsoft/pg_durable/issues
Re: pg_durable: Microsoft open sources in-database durable execution
#48I'm trapped on Azure at work and we're constantly waiting for Azure pg to catch up with modernity. For example, you cant use this: https://www.paradedb.com/blog/hybrid-search-in-postgresql-th... Also for example, you dont get ultra-wide high dimensionality vectors. It is nice they are open sourcing pg_durable, but how about adopting table stakes I'd get with AWS?
ParadeDB is AGPL so not generally available on the hyperscalars. However, you can use https://github.com/timescale/pg_textsearch on Azure HorizonDB (and likely soon Flex). Disclosure: I'm the pg_textsearch maintainer and now at Azure.
I didn't quite follow your comment about vector support, are you asking for something beyond what pgvector + diskann provide (both available on Azure)?
Re: pg_durable: Microsoft open sources in-database durable execution
#49If understanding correctly, Absurd (by the Pi LLM harness devs) minimizes the pure db approach as much as possible. I only just started getting into the topic myself, though. https://github.com/earendil-works/absurd
a nitpick: absurd seems to be an original earendil project they started before Mario Zechner joined earendil, I don't see him in the commits too
but I might not know all the details, I'm genuinely curious
Re: pg_durable: Microsoft open sources in-database durable execution
#50I'm trapped on Azure at work and we're constantly waiting for Azure pg to catch up with modernity. For example, you cant use this: https://www.paradedb.com/blog/hybrid-search-in-postgresql-th... Also for example, you dont get ultra-wide high dimensionality vectors. It is nice they are open sourcing pg_durable, but how about adopting table stakes I'd get with AWS?
Wouldn't Azure Cosmos DB be better suited for vector searches?
no - locking yourself into proprietary single-vendor solutions is never a better option