Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

pg_durable: Microsoft open sources in-database durable execution

github.com

Post reply on HN
Print view View original

101–110 of 119 posts

Re: pg_durable: Microsoft open sources in-database durable execution

#101

Post by TuringNYC »

tjgreen wrote: post #48
TuringNYC wrote: post #22

I'm trapped on Azure at work and we're constantly waiting for Azure pg to catch up with modernity. For example, you cant use this: https://www.paradedb.com/blog/hybrid-search-in-postgresql-th... Also for example, you dont get ultra-wide high dimensionality vectors. It is nice they are open sourcing pg_durable, but how about adopting table stakes I'd get with AWS?

ParadeDB is AGPL so not generally available on the hyperscalars. However, you can use https://github.com/timescale/pg_textsearch on Azure HorizonDB (and likely soon Flex). Disclosure: I'm the pg_textsearch maintainer and now at Azure. I didn't quite follow your comment about vector support, are you asking for something beyond what pgvector + diskann provide (both available on Azure)?

>> I didn't quite follow your comment about vector support, are you asking for something beyond what pgvector + diskann provide (both available on Azure)?

You dont support ultra-wide vectors from the largest embeddings models. We have to wierd stuff like chop up vectors across fields.

Re: pg_durable: Microsoft open sources in-database durable execution

#102

Post by dpark »

mattdeboard wrote: post #100
dpark wrote: post #77

Earlier quoted context omitted.

> This smells like stored procedures. You can’t unit test it. You can’t version it Say what? Stored procedures are awesome when used correctly. Versioning is straightforward. You stick any sort of monotonically increasing id at the end of the name. Whenever you need a breaking change, you bump the id. You also leave the old version with the old id, retiring it only after it’s no longer used. You do need a real story…

The road to eternal burning hell is paved with stored procedures. My experiences (!!) make it so i will never be convinced on the risk:reward being worth it.

What experiences did you have that led you to this conclusion to?

I get the distinct impression that many teams have very weak engineering rigor around their DBs, which leads to a lot of avoidable pain.

Re: pg_durable: Microsoft open sources in-database durable execution

#104

Post by abeomor »

jbonatakis wrote: post #73
abeomor wrote: post #35

Earlier quoted context omitted.

Hey! I'm a PM on the Azure PG team and work on AI features on Postgres. Wanted to address your points directly because we actually ship the capabilities you're asking about, we have made ALOT of progress in the last 3-6 months: Hybrid search (BM25 + vector): Worth noting that ParadeDB's pg_search isn't an AWS-native feature either, you'd need to self-host it on EC2. On Azure PostgreSQL, we built pg_textsearch which p…

> we built pg_textsearch Maybe you meant to word this differently and I’m nitpicking, but didn’t TJ Green build this while he was still at Tiger Data?

Great call out! I meant to say we built support for pg_textsearch extension on HorizonDB

Re: pg_durable: Microsoft open sources in-database durable execution

#105

Post by advertum »

efitz wrote: post #91

My only concern is that AI agents won’t be good at this. For better or worse, they “understand” and have seen a lot of message queuing code and read lots of message queue support discussions.

Agreed, but I think the hard part is the syntax, not the idea. The concept is simple. The way the SQL is written here is unusual, and since there is little training data on it, a model will likely fall back on a more common approach it has seen before.

Re: pg_durable: Microsoft open sources in-database durable execution

#106

Post by SoftTalker »

giancarlostoro wrote: post #37
levkk wrote: post #2

2026 is the year of the Postgres queue! (DBOS[0], pgQue[1]) It's awesome that the community is contributing this and giving us the option to use it. As an ex-app engineer though, I kind of prefer my queue logic to be in code, in Git, but maybe with the right tooling, you can change my mind. :) [0]: https://www.dbos.dev/ [1]: https://github.com/NikolayS/pgque

> As an ex-app engineer though, I kind of prefer my queue logic to be in code, in Git, but maybe with the right tooling, you can change my mind. :) I mean, we used to keep our SQL code in git too for projects where we had DB triggers. I think some were even shoved in there via Django migrations just to let someone setup locally and have the triggers available in their local database.

I've never heard of not source controlling stored procedures, functions, triggers, etc. I source-control all my schema objects, never imagined this isn't normal.

Re: pg_durable: Microsoft open sources in-database durable execution

#107

Post by pokstad »

kilobaud wrote: post #4

> When not to use it > … > The workflow mostly lives outside Postgres and spans many heterogeneous systems. How is this project at all comparable to something like Temporal? Am I misunderstanding the limitation implied by this particular recommendation?

I guess it depends on whether you want to write application code with the Temporal SDK or use this new SQL soup. I’d rather stay out of messy SQL land for something like this if I can avoid it, but I can see the value if you already have Postgres and don’t want to introduce another component.

Re: pg_durable: Microsoft open sources in-database durable execution

#108

Post by otterley »

dpark wrote: post #89
otterley wrote: post #86

Earlier quoted context omitted.

Why would using a stored procedure reduce I/O? I can see it reducing network round trips, but not storage reads and writes.

I was referring to network I/O. But disk I/O should be at least as good for a stored procedure and often better. It’s classic “bring the computation to the data”. Putting the computation into the DB means that use of disk caching (up to and including CPU caching) is maximized. Pulling the data out of the DB to do computation in a higher layer cannot be more efficient in terms of any I/O unless your stored procedure i…

Agreed.

Re: pg_durable: Microsoft open sources in-database durable execution

#110

Post by 0xCMP »

TuringNYC wrote: post #101
tjgreen wrote: post #48

Earlier quoted context omitted.

ParadeDB is AGPL so not generally available on the hyperscalars. However, you can use https://github.com/timescale/pg_textsearch on Azure HorizonDB (and likely soon Flex). Disclosure: I'm the pg_textsearch maintainer and now at Azure. I didn't quite follow your comment about vector support, are you asking for something beyond what pgvector + diskann provide (both available on Azure)?

>> I didn't quite follow your comment about vector support, are you asking for something beyond what pgvector + diskann provide (both available on Azure)? You dont support ultra-wide vectors from the largest embeddings models. We have to wierd stuff like chop up vectors across fields.

Some thing I've learned, but rarely seen explained anywhere: Storing the vectors is most likely not an issue, mostly likely you're having a problem with the indexes on top of them in which case you can use quantized vector indexes[0] (handled by pgvector) which will get past the limits imposed by PostgreSQL.

I had to switch off pgvecto.rs at some point and figured that out.

I don't have specific experience with the Azure environment here, but this probably applies if you have access to pgvector.

[0]: Types of indexes + number of bits supported at bottom of this section: https://github.com/pgvector/pgvector#hnsw

Post reply on HN