Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

pg_durable: Microsoft open sources in-database durable execution

github.com

Post reply on HN
Print view View original

71–80 of 119 posts

Re: pg_durable: Microsoft open sources in-database durable execution

#71

Post by CuriouslyC »

levkk wrote: post #2

2026 is the year of the Postgres queue! (DBOS[0], pgQue[1]) It's awesome that the community is contributing this and giving us the option to use it. As an ex-app engineer though, I kind of prefer my queue logic to be in code, in Git, but maybe with the right tooling, you can change my mind. :) [0]: https://www.dbos.dev/ [1]: https://github.com/NikolayS/pgque

You're not wrong, I've been a big postgres fanboy since version 7, and have tried to build stuff in PG to the greatest degree possible in experiments, and my experience is that at a minimum, the DX/observability isn't there. The multi-master scaling story isn't turnkey or bulletproof either, so I'm hesitant to do any fancy write-bound things that hasten the need to scale the database.

Re: pg_durable: Microsoft open sources in-database durable execution

#72

Post by CuriouslyC »

snqb wrote: post #49
jraedisch wrote: post #7

If understanding correctly, Absurd (by the Pi LLM harness devs) minimizes the pure db approach as much as possible. I only just started getting into the topic myself, though. https://github.com/earendil-works/absurd

a nitpick: absurd seems to be an original earendil project they started before Mario Zechner joined earendil, I don't see him in the commits too but I might not know all the details, I'm genuinely curious

You could call Armin a Pi dev in all honesty. He has a fair number of commits.

Re: pg_durable: Microsoft open sources in-database durable execution

#73

Post by jbonatakis »

abeomor wrote: post #35
TuringNYC wrote: post #22

I'm trapped on Azure at work and we're constantly waiting for Azure pg to catch up with modernity. For example, you cant use this: https://www.paradedb.com/blog/hybrid-search-in-postgresql-th... Also for example, you dont get ultra-wide high dimensionality vectors. It is nice they are open sourcing pg_durable, but how about adopting table stakes I'd get with AWS?

Hey! I'm a PM on the Azure PG team and work on AI features on Postgres. Wanted to address your points directly because we actually ship the capabilities you're asking about, we have made ALOT of progress in the last 3-6 months: Hybrid search (BM25 + vector): Worth noting that ParadeDB's pg_search isn't an AWS-native feature either, you'd need to self-host it on EC2. On Azure PostgreSQL, we built pg_textsearch which p…

> we built pg_textsearch

Maybe you meant to word this differently and I’m nitpicking, but didn’t TJ Green build this while he was still at Tiger Data?

Re: pg_durable: Microsoft open sources in-database durable execution

#74

Post by booi »

sgarland wrote: post #66
faxmeyourcode wrote: post #3

This feels like the wrong solution to an age old problem solved by the DAG schedulers like Apache Airflow for a while now. Why would I want to store my control flow in the database and not in code? It feels strange. Not trying to dismiss the project, I'm just not getting it yet I think.

For one, Airflow (or anything external, for that matter) has no insight into DB load, so when devs slam 200 concurrent workers at the DB, other workloads may be impacted. In contrast, this could (I don’t think it does at this time) get near realtime feedback on performance without the RTT cost, and adjust itself accordingly.

it also feels strange to query for DB load before starting a job.. i'm not even sure how you would do it, how you would adjust a job given a load value, and what would you do if there's too much load.

Re: pg_durable: Microsoft open sources in-database durable execution

#75

Post by Kaliboy »

sgarland wrote: post #67
guhidalg wrote: post #43

Earlier quoted context omitted.

I bet this is correlated with how much they like/know Postgres already. When people don’t understand their database’s features, they want it to behave like something else they do understand (code). They’re leaving a lot of performance on the table by not leveraging everything their database can do.

Yup. Anymore, “we’ll handle that in code” reads to me as “I don’t understand my tools, and don’t want to learn.” Or hubris. The sheer number of times I’ve seen data integrity errors because someone didn’t think they needed a database-level constraint is too damn high. The other one (also related) is normalization. They’ll have hundreds of millions of rows of duplicated, low-cardinality strings, because “joins are exp…

Yeah but the increased I/O is cheaper. It's easier to add another webserver as opposed to upgrading your db server.

And I don't think it's as simple as you make it. So where I work we use amongst other things Rails. There are places in our codebase where using joins just isn't feasable cause the database would use too much memory and let's just say N is large.

But since we use Rails we can have it query the tables apart and join the tables through the defined model. We literally save like 20 seconds in some cases because 1 HUGE query becomes 8 straightforward ones with maximum index usage.

And because we have this capability in Rails we would never use something like this, cause that would neccesate us holding two mental models and have a clear "what do we run where" directive which honestly is a PTA.

Re: pg_durable: Microsoft open sources in-database durable execution

#76

Post by junto »

This smells like stored procedures. You can’t unit test it. You can’t version it. Business logic in the database, (hidden brain problem), harder to isolate noisy workloads, no observability, scaling pressure lands solely in Postgres, lack of IO, especially API calls.

Good for local database only jobs though. Niche use cases.

Re: pg_durable: Microsoft open sources in-database durable execution

#77

Post by dpark »

junto wrote: post #76

This smells like stored procedures. You can’t unit test it. You can’t version it. Business logic in the database, (hidden brain problem), harder to isolate noisy workloads, no observability, scaling pressure lands solely in Postgres, lack of IO, especially API calls. Good for local database only jobs though. Niche use cases.

> This smells like stored procedures. You can’t unit test it. You can’t version it

Say what? Stored procedures are awesome when used correctly.

Versioning is straightforward. You stick any sort of monotonically increasing id at the end of the name. Whenever you need a breaking change, you bump the id. You also leave the old version with the old id, retiring it only after it’s no longer used. You do need a real story for DB upgrades for this to work well. If your story is that someone on the team executes some random SQL migration as root, you’re gonna have a bad time.

You can unit test stored procedures in exactly the same way you could test any other SQL. You have to spin up a DB to do it. But if you can’t test your stored procedures, you’re admitting you have no way to test your SQL which is your real problem.

> Business logic in the database, (hidden brain problem)

Ok? How much you shove into your stored procedures is up to you. In my experience the real alternative to stored procedures is not zero business logic in the DB. It’s SQL code sprinkled throughout the codebase, where it’s harder to test, poorly versioned, and poorly encapsulated. And also often needlessly slow.

> harder to isolate noisy workloads

Dunno what this means

> no observability

Maybe some truth here. It is more work to inspect issues in SQL than most programming languages.

> scaling pressure lands solely in Postgres, lack of IO, especially API calls.

If stored procedures are causing IO problems and scaling issues then you are using them wrong.

Stored procedures often drastically reduce IO when used correctly and thereby improve scalability.

Re: pg_durable: Microsoft open sources in-database durable execution

#78

Post by setr »

Kaliboy wrote: post #75
sgarland wrote: post #67

Earlier quoted context omitted.

Yup. Anymore, “we’ll handle that in code” reads to me as “I don’t understand my tools, and don’t want to learn.” Or hubris. The sheer number of times I’ve seen data integrity errors because someone didn’t think they needed a database-level constraint is too damn high. The other one (also related) is normalization. They’ll have hundreds of millions of rows of duplicated, low-cardinality strings, because “joins are exp…

Yeah but the increased I/O is cheaper. It's easier to add another webserver as opposed to upgrading your db server. And I don't think it's as simple as you make it. So where I work we use amongst other things Rails. There are places in our codebase where using joins just isn't feasable cause the database would use too much memory and let's just say N is large. But since we use Rails we can have it query the tables ap…

> literally save like 20 seconds in some cases because 1 HUGE query becomes 8 straightforward ones with maximum index usage.

I don’t understand how splitting a query up would have any relationship to index utilization; the planner should trivially pick up on it?

Also are you sure you’re not solving a different problem[0]? Doing joins manually being faster doesn’t smell right, except in the case of data duplication increasing total resultset size substantially

Like the cost of increased network load from not filtering through the join should outweigh anything else in the equation

https://learn.microsoft.com/en-us/ef/core/querying/single-sp...

Re: pg_durable: Microsoft open sources in-database durable execution

#79

Post by tmpz22 »

levkk wrote: post #2

2026 is the year of the Postgres queue! (DBOS[0], pgQue[1]) It's awesome that the community is contributing this and giving us the option to use it. As an ex-app engineer though, I kind of prefer my queue logic to be in code, in Git, but maybe with the right tooling, you can change my mind. :) [0]: https://www.dbos.dev/ [1]: https://github.com/NikolayS/pgque

Do you thank the OSS community or Claude?

Re: pg_durable: Microsoft open sources in-database durable execution

#80

Post by sbuttgereit »

steno132 wrote: post #58
reactordev wrote: post #52

Earlier quoted context omitted.

you might be happy to note there is such a thing. pgrust.

This is a great initiative. Postgres was written in the 1980s and we can't afford to have our most utilized workloads running on a software written before most of us even existed. LLMs make it possible to rewrite Postgres and we should take that chance.

> Postgres was written in the 1980s

This is a pretty poor take. Sure the software that we call "PostgreSQL" started to be developed in the 80's... but they didn't stop there. PostgreSQL has been in continuous development, including improvements, changes, and additions, and by some very smart people at that. It's not static and as long as I've been a professional user of the database, decades, it has continually evolved and in some cases even led the way. If we were to survey the software, wouldn't you at least be interested to know how much of code base actually dates back to those long ago decades and how much is more modern before making such statement?

It would be a mistake to take what PostgreSQL actually offers: an excellent database that has be continuously developed and updated over many years (i.e. "maturity"), for some arbitrary idea and evidently baseless idea that somehow "new" must be better.

If new is better, say why; and do so with more actually true statements than it's not extensible. Want it in rust? Well, OK, sure you can give hand-wavy reasons about security and such for why that might be beneficial; but if you want to be convincing you need to be much more specific about the problem in PostgreSQL and the specific way in which your recommendation actually and convincingly moves the needle. If you can't do that, you're simply giving us an emotional outpouring rather than a rational one.

Post reply on HN