When I was a teenager, my uncle had a Kuvasz. She was the most loving dog I ever met, and would easily win a Miss Congeniality contest against any Golden Retriever.
Kuvasz-streamer: open-source CDC for Postgres for low latency replication
31–40 of 51 posts
Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication
#32One will want to be cognizant of its AGPLv3 license https://github.com/kuvasz-io/kuvasz-streamer/blob/v1.19.2/LI...
Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication
#33Good to see another Postgres CDC solution. I have used both Debezium and PeerDB before. Currently, I am using PeerDB in my work to replicate data to ClickHouse and have been loving the experience. The pace at which it performs the initial load is impressive, and the replication latency to ClickHouse is just a few seconds. I haven’t tested PeerDB other targets, such as Kafka, where the latency might be lower.
Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication
#34I like the "type=history" mode which can auto-build a slowly changing dimension ("SCD type 2") for you; more CDC solutions should do that:
https://streamer.kuvasz.io/streaming-modes/
That said, their implementation is kinda poor since it allows overlapping dates for queries when a row gets updated multiple times per day. When you SQL join to that kind of SCD2 by a given date you can easily get duplicates.
This can be avoided by A) updating old rows to end-date yesterday rather than today, and B) if a row begins and ends on the same day, the start date or end date can be NULL or a hardcoded ancient or far-future end-date, such as having the record from "2023-01-01 to 2023-01-01" instead be "2023-01-01 to 0001-01-01". Those rows won't show up in joins, but the change remains visible/auditable and you do get the last row available for every given date (and only one such row.)
Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication
#35What kind of delivery guarantees does this offer? And does it provide data replay? Currently evaluating https://sequinstream.com/ which claims sub-200ms latency, but has a lot of extras that I don’t need and a lighter weight alternative would be nice.
I looked at sequin just now and wish they published their benchmark code. I’m curious how they configure Debezium. At 500 change/s, I get ~90ms average latency with my Debezium cluster after fiddling with Debezium options a bunch. I don’t love Debezium but I also don’t love Erlang, plenty of us have scars from RabbitMQ’s weird sharp edges…
Sequin engineer here. We'll publish our benchmark repo soon! Indeed, we're still doing a lot of fiddling with Debezium ourselves to make sure we cover different configurations, deployments, etc.
The main thing we want to communicate is that we're able to keep up with workloads Debezium can handle.
(And, re: RabbitMQ, I wouldn't write off a platform based on a single application built on that platform :) )
Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication
#36I’ve recently looked into tools like that, I have a busy Postgres table that has a lot of updates on one column and it’s overwhelming Debezium. I’ve tried many things and looked into excluding them from replication with a publication filter but this still causes “events”. Anyone has some pointers on CDC on busy tables?
That’s one of the cases where query-based CDC may out perform log-based (as long as you don’t care to see every intermediate change that happened to a row between syncs)
Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication
#37One will want to be cognizant of its AGPLv3 license https://github.com/kuvasz-io/kuvasz-streamer/blob/v1.19.2/LI...
This is becoming pretty standard for these types of projects that want to be open source but don't want to end up finding out their product got sucked up by AWS and friends. Usually, they offer a commercial license as well.
Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication
#38Earlier quoted context omitted.
Sequinstream is written in Elixir and also pretty recent. https://github.com/sequinstream/sequin Any reason we're seeing so many CDC tools pop up?
The status quo tool Debezium is annoying/heavy because it’s a banana that comes attached to the Java, Kafka Connect, Zookeeper jungle - it’s a massive ecosystem and dependency chain you need to buy into. The Kafka clients outside of Java-land I’ve looked are all sketchy - in Node, KafkaJS went in maintained for years, Confluent recently started maintaining rdkafka-based client that’s somehow slower than the pure JS o…
BTW I think most the ecosystem has settled on https://github.com/twmb/franz-go being the best and highest performing kafka client for Golang (purego)
Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication
#39Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication
#40One will want to be cognizant of its AGPLv3 license https://github.com/kuvasz-io/kuvasz-streamer/blob/v1.19.2/LI...
This is becoming pretty standard for these types of projects that want to be open source but don't want to end up finding out their product got sucked up by AWS and friends. Usually, they offer a commercial license as well.
It should have always been standard, but corps like AWS managed to convince developers to give them free labour, for a while.