One will want to be cognizant of its AGPLv3 license https://github.com/kuvasz-io/kuvasz-streamer/blob/v1.19.2/LI...
why would you want to keep infra helper code private anyway? it's this core for your business?
Kuvasz-streamer: open-source CDC for Postgres for low latency replication
41–50 of 51 posts
Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication
#42lots of Go based CDC stuff going on these days. Redpanda Connect (formerly benthos) recently added support for Postgres CDC [1] and MySQL is coming soon too [2]. 1: https://github.com/redpanda-data/connect/pull/2917 2: https://github.com/redpanda-data/connect/pull/3014
Sequinstream is written in Elixir and also pretty recent. https://github.com/sequinstream/sequin Any reason we're seeing so many CDC tools pop up?
>Any reason we're seeing so many CDC tools pop up?
When I looked for something ~1 year ago to dump to S3 (object storage) they all sucked in some way.
I'm also of the opinion Postgres gives you a pretty "raw" interface with logical replication so a decent amount of building is needed and each person is going to have slightly different requirements/goals.
I haven't looked recently but hopefully these do a better job handling edge cases like TOASTd values, schema changes, and ideally full load
Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication
#43Does anyone know a battle-tested tool that would help with (almost)online migrations of postgresql servers to other hosts? I know it can be done by manually, but I'd like to avoid that
In my experience, everyone's setup is slightly different so it's hard to find a generic solution. pgcopydb is pretty good
I can't remember the name but I saw a Ruby based tool on Hacker News a few months ago that'd automate logical rep setup and failover for you
Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication
#44Are there any benchmarks documented for this, possibly comparing it to alternatives like pgstream or debezium? The "Test report" link on the website[0] returns a 404. [0]: https://streamer.kuvasz.io/report.html
The load test report is here.
Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication
#45I’ve recently looked into tools like that, I have a busy Postgres table that has a lot of updates on one column and it’s overwhelming Debezium. I’ve tried many things and looked into excluding them from replication with a publication filter but this still causes “events”. Anyone has some pointers on CDC on busy tables?
kuvasz-streamer batches updates in a single timed transaction (1 second for example). This is independent of the source transaction.
I have seen a significant increase in performance with this feature.
Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication
#46I like the "type=history" mode which can auto-build a slowly changing dimension ("SCD type 2") for you; more CDC solutions should do that: https://streamer.kuvasz.io/streaming-modes/ That said, their implementation is kinda poor since it allows overlapping dates for queries when a row gets updated multiple times per day. When you SQL join to that kind of SCD2 by a given date you can easily get duplicates. This can be…
The date fields are actually timestamps having microsecond accuracy.
Maybe this was not clear in the docs.
1 | 12 | 1 | r1 | | 1900-01-01 00:00:00+00 | 2025-01-07 21:15:49.233384+00 | f
7 | 12 | 2 | r2 | | 1900-01-01 00:00:00+00 | 2025-01-07 21:15:49.233384+00 | f
13 | 12 | 1 | x1 | | 2025-01-07 21:15:49.233384+00 | 9999-01-01 00:00:00+00 | fRe: Kuvasz-streamer: open-source CDC for Postgres for low latency replication
#47What kind of delivery guarantees does this offer? And does it provide data replay? Currently evaluating https://sequinstream.com/ which claims sub-200ms latency, but has a lot of extras that I don’t need and a lighter weight alternative would be nice.
No guarantees on latency as this depends on the hardware.
Check the load test
Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication
#48Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication
#49Seems very useful. This stuff can’t be done already with pg replication?
I think it’s targeted at [many -> one] database consolidation versus Postgres replication which is more suited for [one -> one]. I’m sure you can do [many -> one] with just Postgres and some shenanigans but it’s probably quite Rube Goldberg. Also in Postgres <16 you can’t replica of replica, using a CDC tool to pipe data around you don’t have that restriction.
It also takes care of setting up publications, replication slots and the initial sync.
Re: Kuvasz-streamer: open-source CDC for Postgres for low latency replication
#50I like the "type=history" mode which can auto-build a slowly changing dimension ("SCD type 2") for you; more CDC solutions should do that: https://streamer.kuvasz.io/streaming-modes/ That said, their implementation is kinda poor since it allows overlapping dates for queries when a row gets updated multiple times per day. When you SQL join to that kind of SCD2 by a given date you can easily get duplicates. This can be…
The date fields are actually timestamps having microsecond accuracy. Maybe this was not clear in the docs. 1 | 12 | 1 | r1 | | 1900-01-01 00:00:00+00 | 2025-01-07 21:15:49.233384+00 | f 7 | 12 | 2 | r2 | | 1900-01-01 00:00:00+00 | 2025-01-07 21:15:49.233384+00 | f 13 | 12 | 1 | x1 | | 2025-01-07 21:15:49.233384+00 | 9999-01-01 00:00:00+00 | f
Ah yeah, makes sense. The docs are misleading.
The tricky thing with DateTime SCD2s vs Date-only SCD2s is that DateTime SCD2s work for identifying what was true for a given click/transaction/ingest time, but doesn't work for identifying what was the final truth associated for a given "business date" such as an "invoice date". That tends to take an ETL or SQL window functions complexity/performance hit. But with streams/CDC, DateTime SCD2s are the easy+clean thing to implement.
Do you use the DateTime the message is received on the target system, or some DateTime from the originating journal/WAL-log?