Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

pg_durable: Microsoft open sources in-database durable execution

github.com

Post reply on HN
Print view View original

31–40 of 119 posts

Re: pg_durable: Microsoft open sources in-database durable execution

#31

Post by gdecandia »

joelthelion wrote: post #24

Isn't the database already one of the hardest piece of infras to scale? Why would you want to load it with additional long-running jobs?

Long-running jobs on Postgres are not new at all. See pg_cron for one example. At the end of the day, these workloads would be running anyway against the database, whether triggered by an external component. HTTP queries from the database have also become more popular to avoid round-trips and failure points from additional components in data or AI pipelines. But yes, whether to bring the compute to the data or vice-versa is a design choice that has a lot of contention.

Re: pg_durable: Microsoft open sources in-database durable execution

#32

Post by Onavo »

gdecandia wrote: post #12
faxmeyourcode wrote: post #6

Earlier quoted context omitted.

I aggree - I'm not understanding the value of the project either if you look at the example here https://github.com/microsoft/pg_durable/blob/main/examples/i... It's an interesting technical achievement I guess, but it's very bizarre to try and read this SELECT df.start( @> ( ($$SELECT ... FROM demo.invoices WHERE status = 'pending'$$ |=> 'inv') ~> df.if_rows('inv', $$UPDATE ... SET status = 'processing'$$ ~> (df.htt…

Contributor here - at Microsoft we've built AI workflows on pg_durable and seen it substantially reduce code and increase reliability. Agree that the DSL ergonomics can be improved. Our pipelines use a higher level language and therefore simplified, but pg_durable is meant to solve a wider array of problems. We're happy to take suggestions for improvements.

Do you plan to open source the high level wrapper too?

Re: pg_durable: Microsoft open sources in-database durable execution

#34

Post by moomoo11 »

levkk wrote: post #2

2026 is the year of the Postgres queue! (DBOS[0], pgQue[1]) It's awesome that the community is contributing this and giving us the option to use it. As an ex-app engineer though, I kind of prefer my queue logic to be in code, in Git, but maybe with the right tooling, you can change my mind. :) [0]: https://www.dbos.dev/ [1]: https://github.com/NikolayS/pgque

same but this could be useful for db level things that are not business logic related.

i have always had maintenance packages for this type of stuff. if i could deploy them alongside the database itself that could be kind of cool.

but yeah i agree with you that i do prefer having this in the code layer.

Re: pg_durable: Microsoft open sources in-database durable execution

#35

Post by abeomor »

TuringNYC wrote: post #22

I'm trapped on Azure at work and we're constantly waiting for Azure pg to catch up with modernity. For example, you cant use this: https://www.paradedb.com/blog/hybrid-search-in-postgresql-th... Also for example, you dont get ultra-wide high dimensionality vectors. It is nice they are open sourcing pg_durable, but how about adopting table stakes I'd get with AWS?

Hey! I'm a PM on the Azure PG team and work on AI features on Postgres. Wanted to address your points directly because we actually ship the capabilities you're asking about, we have made ALOT of progress in the last 3-6 months:

Hybrid search (BM25 + vector): Worth noting that ParadeDB's pg_search isn't an AWS-native feature either, you'd need to self-host it on EC2. On Azure PostgreSQL, we built pg_textsearch which provides the same BM25 ranking model (term frequency saturation, document-length normalization, IDF) natively. Fun fact, the main contributor of pg_textsearch is now on the Azure Postgres team :)

Docs: https://learn.microsoft.com/en-us/azure/horizondb/ai/full-te...

High-dimensional vectors: This is actually an area where we're ahead. pgvector with HNSW caps at 2,000 dimensions. We support pgvector for vector storage and search, and for high-dimensional / large-scale workloads we ship pg_diskann — Microsoft's graph-based vector index that supports up to 16,000 dimensions and also does advanced in-index filtering (your WHERE clauses get evaluated during graph traversal, so you don't lose recall on selective predicates).

pgvector: https://learn.microsoft.com/en-us/azure/horizondb/ai/vector-...

DiskANN high-dimension support: https://learn.microsoft.com/en-us/azure/horizondb/ai/vector-...

These are available today on Azure PostgreSQL, specifically Azure HorizonDB (Preview). Happy to dig into specifics if you have a particular workload in mind.

Re: pg_durable: Microsoft open sources in-database durable execution

#36

Post by affandar »

daxfohl wrote: post #25
faxmeyourcode wrote: post #3

This feels like the wrong solution to an age old problem solved by the DAG schedulers like Apache Airflow for a while now. Why would I want to store my control flow in the database and not in code? It feels strange. Not trying to dismiss the project, I'm just not getting it yet I think.

Microsoft has their own Durable Task framewor[1] for that kind of stuff, and it supports both running as a self-hosted standalone service like temporal, and running serverless on Azure Functions. It actually predated airflow, temporal, etc., IIRC. This one seems to be more database-specific use case. The advantage is probably that you can track the exact state of the job in the database itself, rather than having to…

(Author of both durable task framework and pg_durable/duroxide here)

Indeed Durable tasks is an exceptional project and was a unique innovation at the time.

pg_durable brings the same reliability and durablity semantics to long running operations within the database.

We have tons of interesting scenarios on the roadmap. Stay tuned! :)

Re: pg_durable: Microsoft open sources in-database durable execution

#37

Post by giancarlostoro »

levkk wrote: post #2

2026 is the year of the Postgres queue! (DBOS[0], pgQue[1]) It's awesome that the community is contributing this and giving us the option to use it. As an ex-app engineer though, I kind of prefer my queue logic to be in code, in Git, but maybe with the right tooling, you can change my mind. :) [0]: https://www.dbos.dev/ [1]: https://github.com/NikolayS/pgque

> As an ex-app engineer though, I kind of prefer my queue logic to be in code, in Git, but maybe with the right tooling, you can change my mind. :)

I mean, we used to keep our SQL code in git too for projects where we had DB triggers. I think some were even shoved in there via Django migrations just to let someone setup locally and have the triggers available in their local database.

Re: pg_durable: Microsoft open sources in-database durable execution

#39

Post by gdecandia »

rastignack wrote: post #18

I hope it could be used in the future to export pg_dump formated exports to s3. One would be able to trigger maintenance jobs via simple lambda functions whose duration is capped.

Committer here. I would love to hear more about this scenario.

Is the proposal to be able to export pg_dump formatted data on some schedule or trigger, entirely hosted in PostgreSQL and with timeouts? There are already extension that can export to blob/file storage and can be combined with pg_durable or pg_cron, so I assume the challenge is pg_dump compatible data export from SQL running in the database?

Re: pg_durable: Microsoft open sources in-database durable execution

#40

Post by moron4hire »

TuringNYC wrote: post #22

I'm trapped on Azure at work and we're constantly waiting for Azure pg to catch up with modernity. For example, you cant use this: https://www.paradedb.com/blog/hybrid-search-in-postgresql-th... Also for example, you dont get ultra-wide high dimensionality vectors. It is nice they are open sourcing pg_durable, but how about adopting table stakes I'd get with AWS?

I'm sorry, I'm sure you've considered this, but why couldn't you create a bare VM with Postgres vCurrent installed?
Post reply on HN