MySQL CDC to BigQuery: what periodic syncs miss, and how binlog avoids it • Hacker News Board

Re: MySQL CDC to BigQuery: what periodic syncs miss, and how binlog avoids it

#11

Post by karakanb » Tue, Aug 25, 2026, 9:18 PM UTC

javier2 wrote: post #8

karakanb wrote: post #4

Disclaimer: I am cofounder of Bruin ( https://github.com/bruin-data/bruin ), we are a competitor to Erathos. It seems like a relatively straightforward marketing article. I was pleasantly surprised to learn about Erathos though, nice product! I am personally not a big fan of CDC in prod. Streaming data movement is generally prone to confusion, and it feeds into bad data patterns like hard deletes without any audit lo…

I have no horse in this race, but CDC sounds more robust? It will capture all changes, regardless of how the application maintainers use the database.

One would think. Conceptually they are nice, practically it gets very dirty across different databases + all the infra around them. My dislike stems from the operational complexity and years of trauma around it.

Re: MySQL CDC to BigQuery: what periodic syncs miss, and how binlog avoids it

#12

Post by dcmerritt » Sat, Aug 29, 2026, 7:42 PM UTC

gpaulbagetti wrote: post #2

Wrote this after seeing the same failure mode too many times: a MySQL sync job that looks fine in the dashboard but has been silently dropping deletes and intermediate updates for months, because it's comparing snapshots instead of reading the binlog. Tried to lay out exactly what has to be true on the MySQL side (row image, binlog_row_value_options, server-id, retention) for CDC to actually be complete, not just "ev…

[flagged]

Re: MySQL CDC to BigQuery: what periodic syncs miss, and how binlog avoids it

#13

Post by gpaulbagetti » Mon, Aug 31, 2026, 8:36 PM UTC

karakanb wrote: post #4

Disclaimer: I am cofounder of Bruin ( https://github.com/bruin-data/bruin ), we are a competitor to Erathos. It seems like a relatively straightforward marketing article. I was pleasantly surprised to learn about Erathos though, nice product! I am personally not a big fan of CDC in prod. Streaming data movement is generally prone to confusion, and it feeds into bad data patterns like hard deletes without any audit lo…

Hey! Glad to hear about Bruin, competition always pushes us to make our product better for our customers :)

Frankly, I am not a big fan of streaming either. But CDC and streaming are two different things. Streaming database replication is generally built on CDC, but you can use CDC with micro-batches too (which is what we do). I wrote about this in another article: https://www.erathos.com/en/blog/cursor-based-sync-vs-change-...

Imo, the key reason for offering CDC is that most databases aren't designed around "how can we extract data efficiently", therefore they often lack a reliable "updated_at" column for us to use as a cursor.