Looks nice! How does this compare to PeerDB?
Kuvasz-streamer: open-source CDC for Postgres for low latency replication
21–30 of 51 posts
Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication
#22What kind of delivery guarantees does this offer? And does it provide data replay?
Currently evaluating https://sequinstream.com/ which claims sub-200ms latency, but has a lot of extras that I don’t need and a lighter weight alternative would be nice.
Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication
#23What kind of delivery guarantees does this offer? And does it provide data replay? Currently evaluating https://sequinstream.com/ which claims sub-200ms latency, but has a lot of extras that I don’t need and a lighter weight alternative would be nice.
I looked at sequin just now and wish they published their benchmark code. I’m curious how they configure Debezium. At 500 change/s, I get ~90ms average latency with my Debezium cluster after fiddling with Debezium options a bunch.
I don’t love Debezium but I also don’t love Erlang, plenty of us have scars from RabbitMQ’s weird sharp edges…
Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication
#24lots of Go based CDC stuff going on these days. Redpanda Connect (formerly benthos) recently added support for Postgres CDC [1] and MySQL is coming soon too [2]. 1: https://github.com/redpanda-data/connect/pull/2917 2: https://github.com/redpanda-data/connect/pull/3014
Sequinstream is written in Elixir and also pretty recent. https://github.com/sequinstream/sequin Any reason we're seeing so many CDC tools pop up?
The status quo tool Debezium is annoying/heavy because it’s a banana that comes attached to the Java, Kafka Connect, Zookeeper jungle - it’s a massive ecosystem and dependency chain you need to buy into. The Kafka clients outside of Java-land I’ve looked are all sketchy - in Node, KafkaJS went in maintained for years, Confluent recently started maintaining rdkafka-based client that’s somehow slower than the pure JS one and breaks every time I try to upgrade it. The Rust Kafka client has months-old issues in the latest release where half the messages go missing and APIs seem to no-op, and any version will SIGSEGV if you hold it wrong - obviously memory unsafe. The official rdkafka Go client depends on system C library package versions “matching up” meaning you often need a newer librdkafka and libsasl which is annoying; the unofficial pure-go one looks decent though.
Overall the Confluent ecosystem feels targeted at “data engineer” use-cases so if you want to build a reactive product it’s not a great fit. I’m not sure what the performance target is of the Debezium Postgres connector maintainers but I get the sense they’re not ambitious because there’s so little documentation about performance optimization; data ecosystem feels contemporary with “nightly batch job” kind of thing vs product people today who want 0ms latency.
If you look at backend infrastructure there’s a clear trope of “good idea implemented in Java becomes standard, but re-implementing in $AOT_COMPILED_LANGUAGE gives big advantage:
- Cassandra -> ScyllaDB
- Kafka -> RedPanda, …
- Zookeeper -> ClickHouse Keeper, Consul, etcd, …
- Debezium -> All these thingies
There’s also a lot of hype around Postgres right now, so a bit of VC funded Cambrian explosion going on and I think a lot these will die off as a clear winner emerges.
Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication
#25What kind of delivery guarantees does this offer? And does it provide data replay? Currently evaluating https://sequinstream.com/ which claims sub-200ms latency, but has a lot of extras that I don’t need and a lighter weight alternative would be nice.
You can get single digit latency with almost anything for CDC to queue if you properly colocate your services & your CDC ingestion is keeping up with postgres slot
Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication
#26Seems very useful. This stuff can’t be done already with pg replication?
I think it’s targeted at [many -> one] database consolidation versus Postgres replication which is more suited for [one -> one]. I’m sure you can do [many -> one] with just Postgres and some shenanigans but it’s probably quite Rube Goldberg. Also in Postgres <16 you can’t replica of replica, using a CDC tool to pipe data around you don’t have that restriction.
Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication
#27Earlier quoted context omitted.
From the newish Go-based Postgres CDC tools, I know about: * pgstream: https://github.com/xataio/pgstream * pg_flo: https://github.com/pgflo/pg_flo Are there others? Each of them has slightly different angles and messaging, but it is interesting to see.
https://github.com/artie-labs/reader is another I know of
No support for streaming PG changes
Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication
#28What kind of delivery guarantees does this offer? And does it provide data replay? Currently evaluating https://sequinstream.com/ which claims sub-200ms latency, but has a lot of extras that I don’t need and a lighter weight alternative would be nice.
You can get single digit latency with almost anything for CDC to queue if you properly colocate your services & your CDC ingestion is keeping up with postgres slot
Should have specified; this is for tracking bursts of up to 1.5k tps not in the same AZ or even VPC.
Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication
#29[flagged]
Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication
#30[flagged]
[deleted]