Live data from Hacker News

MySQL CDC to BigQuery: what periodic syncs miss, and how binlog avoids it

erathos.com

11–12 of 12 posts

Re: MySQL CDC to BigQuery: what periodic syncs miss, and how binlog avoids it

#11
post #8
post #4

Disclaimer: I am cofounder of Bruin ( https://github.com/bruin-data/bruin ), we are a competitor to Erathos. It seems like a relatively straightforward marketing article. I was pleasantly surprised to learn about Erathos though, nice product! I am personally not a big fan of CDC in prod. Streaming data movement is generally prone to confusion, and it feeds into bad data patterns like hard deletes without any audit lo…

I have no horse in this race, but CDC sounds more robust? It will capture all changes, regardless of how the application maintainers use the database.

One would think. Conceptually they are nice, practically it gets very dirty across different databases + all the infra around them. My dislike stems from the operational complexity and years of trauma around it.

Re: MySQL CDC to BigQuery: what periodic syncs miss, and how binlog avoids it

#12

Wrote this after seeing the same failure mode too many times: a MySQL sync job that looks fine in the dashboard but has been silently dropping deletes and intermediate updates for months, because it's comparing snapshots instead of reading the binlog. Tried to lay out exactly what has to be true on the MySQL side (row image, binlog_row_value_options, server-id, retention) for CDC to actually be complete, not just "ev…

[flagged]
Post reply on HN