11–20 of 38 posts
Snowflake is a really amazing product. It's been a delight using it the last few years.
EDIT: I now see it's mainly to do with pushing data out of customer's postgres systems into snowflake
Although pg_lake is open source, worth noting that it heavily refers to but is missing CDC capabilities. There's a bunch of comments/links to a closed https://github.com/snowflake-eng/sfpg-extension-pg_lake_repl...
Although pg_lake is open source, worth noting that it heavily refers to but is missing CDC capabilities. There's a bunch of comments/links to a closed https://github.com/snowflake-eng/sfpg-extension-pg_lake_repl...
I'm interested in pg_lake so I wanted to check out your link, but it seems to be internal to snowflake?
[1] https://github.com/Snowflake-Labs/pg_lake
[2] https://www.snowflake.com/en/blog/engineering/pg-lake-postgr...
- ClickHouse focuses on traditional CDC (ClickPipes) and just make it blazingly fast
- Databricks leans on their unified storage architecture (LTAP) to avoid copying data (though you can argue there is still a copy in the cache)
- Snowflake uses a data mirroring CDC as extension so it runs directly on Postgres
I'm still waiting for a Postgres provider to just let me mirror data directly to Iceberg, so I can plug in my own stateless query engine.
Although pg_lake is open source, worth noting that it heavily refers to but is missing CDC capabilities. There's a bunch of comments/links to a closed https://github.com/snowflake-eng/sfpg-extension-pg_lake_repl...
I'm interested in pg_lake so I wanted to check out your link, but it seems to be internal to snowflake?
The article links https://github.com/Snowflake-Labs/pg_lake but if you go looking for CDC, it's not there, and all you end up with is links to the private/closed repo that I just linked, in various corners and spots. The point is that all the CDC stuff is in the repo we don't get access to, that isn't open source: https://github.com/snowflake-eng/sfpg-extension-pg_lake_repl...
It's interesting to watch how different companies that offer both Postgres and warehousing solution under 1 roof approach the same problem: - ClickHouse focuses on traditional CDC (ClickPipes) and just make it blazingly fast - Databricks leans on their unified storage architecture (LTAP) to avoid copying data (though you can argue there is still a copy in the cache) - Snowflake uses a data mirroring CDC as extension…
The issue that each of those providers above has recently adopted Postgres as a secondary product aimed at supporting their main product, an OLAP database or engine, so they don’t want you plugging in your own query engine.
I’d bet you’re likely to see this from a Postgres-specific provider first, like Supabase.
Snowflake is a really amazing product. It's been a delight using it the last few years.
Seeing Postgres articles from Snowflake surprises me a lot though given how there's zero relation between Snowflake the product and Postgres itself EDIT: I now see it's mainly to do with pushing data out of customer's postgres systems into snowflake
It's interesting to watch how different companies that offer both Postgres and warehousing solution under 1 roof approach the same problem: - ClickHouse focuses on traditional CDC (ClickPipes) and just make it blazingly fast - Databricks leans on their unified storage architecture (LTAP) to avoid copying data (though you can argue there is still a copy in the cache) - Snowflake uses a data mirroring CDC as extension…
There’s not a mirror function like what Snowflake offers directly but you can come close with a pgcron to upsert changes to the iceberg tables every so often.
You can also purge the table put to the iceberg version every so often too depending on your data needs. Then you can create a query unions the results of both.
Snowflake is a really amazing product. It's been a delight using it the last few years.
as much praise as some people give to it, I feel deeply uncomfortable with an idea of SaaS-only DB tech that you don't have an option to self host
It's interesting to watch how different companies that offer both Postgres and warehousing solution under 1 roof approach the same problem: - ClickHouse focuses on traditional CDC (ClickPipes) and just make it blazingly fast - Databricks leans on their unified storage architecture (LTAP) to avoid copying data (though you can argue there is still a copy in the cache) - Snowflake uses a data mirroring CDC as extension…
At Snowflake we use Snowflake to do the apply work. Hence end-to-end mirroring has more pieces than just Postgres, but the capturing of changes is cheap enough to do in Postgres directly.
(Author)