Now fivetran has more competition. Good.
How We Pushed CDC into Postgres
21–30 of 38 posts
Re: How We Pushed CDC into Postgres
#22It's interesting to watch how different companies that offer both Postgres and warehousing solution under 1 roof approach the same problem: - ClickHouse focuses on traditional CDC (ClickPipes) and just make it blazingly fast - Databricks leans on their unified storage architecture (LTAP) to avoid copying data (though you can argue there is still a copy in the cache) - Snowflake uses a data mirroring CDC as extension…
The challenge is converting primary key updates/deletes to row offsets in a columnar table. That requires maintaining an expensive mapping or doing expensive scans, and is not something you want Postgres itself to do. It's also this bursty, memory-intensive workload that you'd rather not have a lot of dedicated infrastructure for. At Snowflake we use Snowflake to do the apply work. Hence end-to-end mirroring has more…
Yes I’m not suggesting to do this inside Postgres. I’m hoping that a Postgres provider can provide this mirroring capability out of the box, similar to how they provide a connection-pooled endpoint out of the box so I don’t have to self-host pgBouncer. I just want to be able to check a box somewhere and have a table in Postgres automatically mirrored to Iceberg, with guarantee that no data is lost. They can charge more for it, I will gladly pay.
Re: How We Pushed CDC into Postgres
#23It's interesting to watch how different companies that offer both Postgres and warehousing solution under 1 roof approach the same problem: - ClickHouse focuses on traditional CDC (ClickPipes) and just make it blazingly fast - Databricks leans on their unified storage architecture (LTAP) to avoid copying data (though you can argue there is still a copy in the cache) - Snowflake uses a data mirroring CDC as extension…
> I'm still waiting for a Postgres provider to just let me mirror data directly to Iceberg, so I can plug in my own stateless query engine. The issue that each of those providers above has recently adopted Postgres as a secondary product aimed at supporting their main product, an OLAP database or engine, so they don’t want you plugging in your own query engine. I’d bet you’re likely to see this from a Postgres-specif…
I’m waiting for Cloudflare R2 to eventually support mirroring Postgres into R2 catalog. It seems like a nice fit because they already have R2 SQL.
Re: How We Pushed CDC into Postgres
#24Excellent that Control Data is contributing! Oh wait, I'm a few decades out of sync
Re: How We Pushed CDC into Postgres
#25It's interesting to watch how different companies that offer both Postgres and warehousing solution under 1 roof approach the same problem: - ClickHouse focuses on traditional CDC (ClickPipes) and just make it blazingly fast - Databricks leans on their unified storage architecture (LTAP) to avoid copying data (though you can argue there is still a copy in the cache) - Snowflake uses a data mirroring CDC as extension…
> I'm still waiting for a Postgres provider to just let me mirror data directly to Iceberg, so I can plug in my own stateless query engine.
If you count AWS as Postgres provider, DMS into Kinesis into Firehose can do that.
There was preview of just Firehose doing it directly, but AWS have pulled it because it was too unreliable. Maybe they rebuilt it since?
Re: How We Pushed CDC into Postgres
#26[flagged]
Re: How We Pushed CDC into Postgres
#27It's interesting to watch how different companies that offer both Postgres and warehousing solution under 1 roof approach the same problem: - ClickHouse focuses on traditional CDC (ClickPipes) and just make it blazingly fast - Databricks leans on their unified storage architecture (LTAP) to avoid copying data (though you can argue there is still a copy in the cache) - Snowflake uses a data mirroring CDC as extension…
Google: Charges out the wazoo for a half baked product built on top of Dataflow
Re: How We Pushed CDC into Postgres
#28Excellent that Control Data is contributing! Oh wait, I'm a few decades out of sync
I'm afraid nobody (else) here remembers Control Data Corporation. But I'm happy to see the Centers for Disease Control using Postgres.
Re: How We Pushed CDC into Postgres
#29Earlier quoted context omitted.
I'm interested in pg_lake so I wanted to check out your link, but it seems to be internal to snowflake?
pg_lake is an open source Postgres extension based on work done at Crunchy Data prior to the acquisition by Snowflake - you can find the repo here [1] and a blog post with more context on the project here [2] [1] https://github.com/Snowflake-Labs/pg_lake [2] https://www.snowflake.com/en/blog/engineering/pg-lake-postgr...
I see you are a co-founder! Thanks for pg_lake. I am actually already heavily using it, was just interested to read about CDC in the context of pg_lake.
Re: How We Pushed CDC into Postgres
#30What did Center for Disease Control use before? Mysql?