Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kuvasz-streamer: open-source CDC for Postgres for low latency replication

streamer.kuvasz.io

Post reply on HN
Print view View original

11–20 of 51 posts

Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication

#11

Post by dikei »

olavgg wrote: post #10

How does it compare versus Debezium?

I'd be more similar to Debezium-server that runs everything in the same process, than regular KafkaConnect-based Debezium.

However, this only does postgres-postgres, so it's a lot more limited compared to Debezium.

Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication

#12

Post by andruby »

rockwotj wrote: post #6

lots of Go based CDC stuff going on these days. Redpanda Connect (formerly benthos) recently added support for Postgres CDC [1] and MySQL is coming soon too [2]. 1: https://github.com/redpanda-data/connect/pull/2917 2: https://github.com/redpanda-data/connect/pull/3014

Sequinstream is written in Elixir and also pretty recent.

https://github.com/sequinstream/sequin

Any reason we're seeing so many CDC tools pop up?

Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication

#13

Post by andruby »

CAP_NET_ADMIN wrote: post #5

Does anyone know a battle-tested tool that would help with (almost)online migrations of postgresql servers to other hosts? I know it can be done by manually, but I'd like to avoid that

PG's internal wal-level replication? primary to a read-replica, and then switch the read to become the primary. You'll have a bit of downtime while you stop connections on the original server, switch the new server to primary, and update your app config to connect to the new server.

I believe that's a pretty standard way to provide "HA" postgres. (We use Patroni for our HA setup)

https://github.com/patroni/patroni

Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication

#14

Post by dikei »

andruby wrote: post #13
CAP_NET_ADMIN wrote: post #5

Does anyone know a battle-tested tool that would help with (almost)online migrations of postgresql servers to other hosts? I know it can be done by manually, but I'd like to avoid that

PG's internal wal-level replication? primary to a read-replica, and then switch the read to become the primary. You'll have a bit of downtime while you stop connections on the original server, switch the new server to primary, and update your app config to connect to the new server. I believe that's a pretty standard way to provide "HA" postgres. (We use Patroni for our HA setup) https://github.com/patroni/patroni

We use the same setup, though we use PGBouncer so after switching primary we just force reconnect all clients from PGBouncer instead.

The clients will have to retry on-going transactions, but that's a basic fault tolerant requirement anyway.

Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication

#15

Post by hans_castorp »

CAP_NET_ADMIN wrote: post #5

Does anyone know a battle-tested tool that would help with (almost)online migrations of postgresql servers to other hosts? I know it can be done by manually, but I'd like to avoid that

pglogical can do that (or at least minimize the manual steps as much as possible)

I am not entirely sure, but I think CloudNativePG (a Kubernetes operator) can also be used for that.

Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication

#17

Post by dewey »

I’ve recently looked into tools like that, I have a busy Postgres table that has a lot of updates on one column and it’s overwhelming Debezium.

I’ve tried many things and looked into excluding them from replication with a publication filter but this still causes “events”.

Anyone has some pointers on CDC on busy tables?

Post reply on HN