Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Peerdb Streams – Simple, native Postgres change data capture

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 42 posts

Re: Show HN: Peerdb Streams – Simple, native Postgres change data capture

#31

Post by saisrirampur »

tarun_anand wrote: post #29

Does peerdb help in replicating from one citus cluster to another citus cluster ? If not, then is there any other solution?

We can use PeerDB to replicate each node (worker and coordinator) from one Citus cluster to corresponding nodes in another Citus cluster. It will require some customization, and I am happy to assist you through the process. Also, just as a heads up, prior to PeerDB, I worked at Citus for 8 years as an early (solutions) engineer, so I have a pretty good understanding of Citus internals. :)

Re: Show HN: Peerdb Streams – Simple, native Postgres change data capture

#32

Post by saisrirampur »

take-five wrote: post #30
saisrirampur wrote: post #11

Earlier quoted context omitted.

Great question! We have retry logic in place to handle Postgres restarts. If the failure occurs in-place, you should be fine as the slot will persist. If Postgres fails over to the standby, PeerDB will wait until the slot is created. We did consider automatically creating the slot if it doesn't exist on retries, but ensuring data reliability (creating the slot right after failover without data being ingested) is tric…

Thanks! I'd say, this is a very complicated problem left to be solved by user. Debezium also does not solve the problem it, and from DBA point of view it is a blocker for adoption. Would be nice to have some solution that would ensure that logical replication slot persists through failover.

Quite a bit of work is going on to support failover slots in upstream - https://github.com/search?q=repo%3Apostgres%2Fpostgres+failo... From what I understood it could get into Postgres 17 (released later this year) if there are no big issues. :)

Re: Show HN: Peerdb Streams – Simple, native Postgres change data capture

#33

Post by jensneuse »

We're currently in the process of adding Kafka support to EDFS (Event Driven Federated Subscriptions), a specification to add NATS, Kafka, and other pub sub or streaming services to a federated graph.

There's one thing missing and we could hook it up to your CDC solution. For each message, we need to set the __typename field. Does your solution have a way to accomplish this?

EDFS reference: https://cosmo-docs.wundergraph.com/router/event-driven-feder...

Re: Show HN: Peerdb Streams – Simple, native Postgres change data capture

#34

Post by saisrirampur »

jensneuse wrote: post #33

We're currently in the process of adding Kafka support to EDFS (Event Driven Federated Subscriptions), a specification to add NATS, Kafka, and other pub sub or streaming services to a federated graph. There's one thing missing and we could hook it up to your CDC solution. For each message, we need to set the __typename field. Does your solution have a way to accomplish this? EDFS reference: https://cosmo-docs.wunderg…

Yes, that is doable through a very simple Lua-based row-level transformation. :) Here is an example transformation that adds an additional field: https://github.com/PeerDB-io/examples/blob/main/usdcad.lua . More details on row-level transformations are planned to be released later this week. :)

Re: Show HN: Peerdb Streams – Simple, native Postgres change data capture

#35

Post by jensneuse »

saisrirampur wrote: post #34
jensneuse wrote: post #33

We're currently in the process of adding Kafka support to EDFS (Event Driven Federated Subscriptions), a specification to add NATS, Kafka, and other pub sub or streaming services to a federated graph. There's one thing missing and we could hook it up to your CDC solution. For each message, we need to set the __typename field. Does your solution have a way to accomplish this? EDFS reference: https://cosmo-docs.wunderg…

Yes, that is doable through a very simple Lua-based row-level transformation. :) Here is an example transformation that adds an additional field: https://github.com/PeerDB-io/examples/blob/main/usdcad.lua . More details on row-level transformations are planned to be released later this week. :)

If you're open to it, we could collaborate on a demo of the two components, PeerDB CDC and EDFS, and work together on cross promoting the solutions. We've got a large customer base in the enterprise segment that's interested in EDFS with Kafka, and I think that your solution fits very well into the picture.

If you're interested, just fill our contact form and mention this thread. This could be really cool!

Re: Show HN: Peerdb Streams – Simple, native Postgres change data capture

#36

Post by gunnarmorling »

take-five wrote: post #30
saisrirampur wrote: post #11

Earlier quoted context omitted.

Great question! We have retry logic in place to handle Postgres restarts. If the failure occurs in-place, you should be fine as the slot will persist. If Postgres fails over to the standby, PeerDB will wait until the slot is created. We did consider automatically creating the slot if it doesn't exist on retries, but ensuring data reliability (creating the slot right after failover without data being ingested) is tric…

Thanks! I'd say, this is a very complicated problem left to be solved by user. Debezium also does not solve the problem it, and from DBA point of view it is a blocker for adoption. Would be nice to have some solution that would ensure that logical replication slot persists through failover.

Things got finally a bit easier with Postgres 16, which allows you to create replication slots on stand-by servers. It still requires manual intervention to make sure they don't fall too far behind their primary counterpart and to do the failover, but it's not too terrible either and it's guaranteed that failover can happen without missing any events. I've described the approach here: https://www.decodable.co/blog/logical-replication-from-postg.... Postgres 17 should make this even simpler.

Re: Show HN: Peerdb Streams – Simple, native Postgres change data capture

#37

Post by zknill »

Why do you recommend a heartbeat table to mitigate WAL slot growth if the PeerDB Stream targets a specific table? Presumably this means that the WAL slot is subscribed to all table changes, even if only specific tables are actually included in the CDC? Why not just subscribe to the WAL that you need?

Re: Show HN: Peerdb Streams – Simple, native Postgres change data capture

#38

Post by nijave »

arsalanb wrote: post #12

Noob question: What is the advantage of replicating data into a warehouse vs. just querying it in place on a postgres database?

Event sourcing

Also reporting/analytics work loads tend to be ad hoc queries and hard to optimize so you generally favor fast storage over indexes. Frequently for analytics and reporting it's more efficient to use a columnar DB than a relational db

Re: Show HN: Peerdb Streams – Simple, native Postgres change data capture

#39

Post by WJW »

vivzkestrel wrote: post #26

what is the difference between using this library and PG_LISTEN/NOTIFY with triggers for each row change?

Triggers have much larger overhead than writing to the replication stream (which most databases should be doing anyway), so if you have a lot of traffic this is faster.

Re: Show HN: Peerdb Streams – Simple, native Postgres change data capture

#40

Post by vivzkestrel »

WJW wrote: post #39
vivzkestrel wrote: post #26

what is the difference between using this library and PG_LISTEN/NOTIFY with triggers for each row change?

Triggers have much larger overhead than writing to the replication stream (which most databases should be doing anyway), so if you have a lot of traffic this is faster.

sorry but didn't understand, what exactly is this replication stream, are you talking about the wal logs? any ideas why postgres hasn't implemented this natively if it is better?
Post reply on HN