pg_durable: Microsoft open sources in-database durable execution • Hacker News Board

Re: pg_durable: Microsoft open sources in-database durable execution

#61

Post by jrumbut » Fri, Jun 05, 2026, 9:10 PM UTC

giancarlostoro wrote: post #37

levkk wrote: post #2

2026 is the year of the Postgres queue! (DBOS[0], pgQue[1]) It's awesome that the community is contributing this and giving us the option to use it. As an ex-app engineer though, I kind of prefer my queue logic to be in code, in Git, but maybe with the right tooling, you can change my mind. :) [0]: https://www.dbos.dev/ [1]: https://github.com/NikolayS/pgque

> As an ex-app engineer though, I kind of prefer my queue logic to be in code, in Git, but maybe with the right tooling, you can change my mind. :) I mean, we used to keep our SQL code in git too for projects where we had DB triggers. I think some were even shoved in there via Django migrations just to let someone setup locally and have the triggers available in their local database.

If you have triggers I don't see why you wouldn't put them in a migration. That addresses one of the most problematic aspects of triggers (invisibility, no version tracking, etc) without reducing their usefulness.

With some cleverness you could even introduce some testing that way. Not perfect but better than nothing.

Re: pg_durable: Microsoft open sources in-database durable execution

#62

Post by dsr_ » Fri, Jun 05, 2026, 9:13 PM UTC

linuxhiker wrote: post #55

steno132 wrote: post #51

I would argue that for all but the largest tech companies you only need a single data system which is Postgres. Message brokers, analytical databases all can be built on Postgres. Unfortunately, Postgres as it's built now lacks any semblence of extensibility which makes this impossible in practice. I would propose a rewrite of Postgres in another language like Rust, introducing a pluggable application layer on top. W…

I am afraid you don't understand Postgres very well.

What's to understand? They think they can vibecode PG19.

I won't be running that, though.

Re: pg_durable: Microsoft open sources in-database durable execution

#63

Post by 7373737373 » Fri, Jun 05, 2026, 9:15 PM UTC

Feels like perhaps yet another https://en.wikipedia.org/wiki/Inner-platform_effect that would be unnecessary if popular programming languages/virtual machines already supported determinism, metered and controllable stepwise execution and runtime state suspension, (de)serialization and resumption?

Re: pg_durable: Microsoft open sources in-database durable execution

#64

Post by ijustlovemath » Fri, Jun 05, 2026, 9:23 PM UTC

We made a very functional job queue in Postgres with PostgREST. highly recommend, as the automatic REST API makes building new clients a breeze

Re: pg_durable: Microsoft open sources in-database durable execution

#65

Post by sgarland » Fri, Jun 05, 2026, 9:51 PM UTC

steno132 wrote: post #58

reactordev wrote: post #52

Earlier quoted context omitted.

you might be happy to note there is such a thing. pgrust.

This is a great initiative. Postgres was written in the 1980s and we can't afford to have our most utilized workloads running on a software written before most of us even existed. LLMs make it possible to rewrite Postgres and we should take that chance.

It’s always been possible to rewrite it; LLMs just made it possible for people who don’t understand it to do so. I personally don’t see that as a benefit, but you do you.

As to the age statement, why on earth does that matter? HAProxy launched in 2001, and despite it being 25 years old, you’ll struggle to find anything faster or more stable in its field. Then there’s, you know, Linux - 1991. I suppose you want to see it rewritten?

Re: pg_durable: Microsoft open sources in-database durable execution

#66

Post by sgarland » Fri, Jun 05, 2026, 9:53 PM UTC

faxmeyourcode wrote: post #3

This feels like the wrong solution to an age old problem solved by the DAG schedulers like Apache Airflow for a while now. Why would I want to store my control flow in the database and not in code? It feels strange. Not trying to dismiss the project, I'm just not getting it yet I think.

For one, Airflow (or anything external, for that matter) has no insight into DB load, so when devs slam 200 concurrent workers at the DB, other workloads may be impacted. In contrast, this could (I don’t think it does at this time) get near realtime feedback on performance without the RTT cost, and adjust itself accordingly.

Re: pg_durable: Microsoft open sources in-database durable execution

#67

Post by sgarland » Fri, Jun 05, 2026, 10:01 PM UTC

guhidalg wrote: post #43

gdecandia wrote: post #14

Earlier quoted context omitted.

Contributor here. At Microsoft, our Postgres customers seem to split pretty evenly into 2 camps, those that want to do as much as they can in the database, and those that agree with your take - want to keep apps and compute outside the DB.

I bet this is correlated with how much they like/know Postgres already. When people don’t understand their database’s features, they want it to behave like something else they do understand (code). They’re leaving a lot of performance on the table by not leveraging everything their database can do.

Yup. Anymore, “we’ll handle that in code” reads to me as “I don’t understand my tools, and don’t want to learn.” Or hubris. The sheer number of times I’ve seen data integrity errors because someone didn’t think they needed a database-level constraint is too damn high.

The other one (also related) is normalization. They’ll have hundreds of millions of rows of duplicated, low-cardinality strings, because “joins are expensive,” while somehow missing the fact that the increased I/O from reduced page-packing also has a performance cost.

Re: pg_durable: Microsoft open sources in-database durable execution

#68

Post by miohtama » Fri, Jun 05, 2026, 10:11 PM UTC

faxmeyourcode wrote: post #6

kilobaud wrote: post #4

> When not to use it > … > The workflow mostly lives outside Postgres and spans many heterogeneous systems. How is this project at all comparable to something like Temporal? Am I misunderstanding the limitation implied by this particular recommendation?

I aggree - I'm not understanding the value of the project either if you look at the example here https://github.com/microsoft/pg_durable/blob/main/examples/i... It's an interesting technical achievement I guess, but it's very bizarre to try and read this SELECT df.start( @> ( ($$SELECT ... FROM demo.invoices WHERE status = 'pending'$$ |=> 'inv') ~> df.if_rows('inv', $$UPDATE ... SET status = 'processing'$$ ~> (df.htt…

Before this I thought it was impossible to surpass Perl.

Re: pg_durable: Microsoft open sources in-database durable execution

#69

Post by jiggawatts » Fri, Jun 05, 2026, 10:34 PM UTC

antonkochubey wrote: post #50

FuriouslyAdrift wrote: post #44

Earlier quoted context omitted.

Wouldn't Azure Cosmos DB be better suited for vector searches?

no - locking yourself into proprietary single-vendor solutions is never a better option

I’m not sure why you’re getting downvoted because CosmosDB doesn’t even have a local install edition. Conversely the cloud hosted offering is slower than cold molasses and costs most of your body parts… per month.

Re: pg_durable: Microsoft open sources in-database durable execution

#70

Post by jiggawatts » Fri, Jun 05, 2026, 10:55 PM UTC

gdecandia wrote: post #33

mikey_p wrote: post #19

Is this an open sourcing of something they use internally? My first thought on durable jobs was GHA aka Azure Devops.

Please see https://learn.microsoft.com/en-us/azure/horizondb/ai/ai-pipe...

These approaches always suffer from the same issues such as synchronous single threaded code that would be trivial to parallelise in a “proper” programming language such as C#.

What has Microsoft done to work around this?