Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire

github.com

Post reply on HN
Print view View original

21–30 of 47 posts

Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire

#21

Post by raducu »

vira28 wrote: post #5

Author here. For context, I was the tech lead for the Postgres team at Cloudflare, and this came directly out of a challenge I kept hitting there: BI and dashboard teams needed to run long-running analytical queries, and the answer was always to spin up another bespoke read replica or stand up an ETL dump into an analytical database and query that. So the question I started with was: what's the fewest components I co…

> queryable from psql via an embedded DuckDB.

noob question here from someone who ony played a bit with iceberg and trino: what's the reason to do the analytics stil inside the postgres -- is it so that you don't eat up the IOPS/bandwidth of the main postgresql disks?

Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire

#22

Post by iamcreasy »

vira28 wrote: post #5

Author here. For context, I was the tech lead for the Postgres team at Cloudflare, and this came directly out of a challenge I kept hitting there: BI and dashboard teams needed to run long-running analytical queries, and the answer was always to spin up another bespoke read replica or stand up an ETL dump into an analytical database and query that. So the question I started with was: what's the fewest components I co…

Very cool! What would a 10,000 feet solution look like for MySQL to Iceberg on S3?

Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire

#24

Post by viveknathani_ »

saisrirampur wrote: post #20
viveknathani_ wrote: post #17

interesting approach, was exploring a Postgres to Clickhouse CDC setup while helping a team sometime back, this seems better as it allows separating the compute (query server) and storage (s3) layers, and thereby allowing us to be creative in cost reductions

It depends on the use case. For real-time, customer-facing analytics, ClickHouse’s MergeTree engine is a natural fit, so a Postgres → ClickHouse CDC setup with low latencies (single-digit seconds) is better. Replication to Iceberg/S3 is better suited for offline analytics and data warehousing use cases. You can use the same ClickHouse engine to query layer Iceberg data in S3.

makes sense!

Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire

#25

Post by vira28 »

iamcreasy wrote: post #22
vira28 wrote: post #5

Author here. For context, I was the tech lead for the Postgres team at Cloudflare, and this came directly out of a challenge I kept hitting there: BI and dashboard teams needed to run long-running analytical queries, and the answer was always to spin up another bespoke read replica or stand up an ETL dump into an analytical database and query that. So the question I started with was: what's the fewest components I co…

Very cool! What would a 10,000 feet solution look like for MySQL to Iceberg on S3?

Should be fairly doable using binlog-based producer https://github.com/go-mysql-org/go-mysql.

Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire

#26

Post by vira28 »

ashtuchkin wrote: post #8
vira28 wrote: post #5

Author here. For context, I was the tech lead for the Postgres team at Cloudflare, and this came directly out of a challenge I kept hitting there: BI and dashboard teams needed to run long-running analytical queries, and the answer was always to spin up another bespoke read replica or stand up an ETL dump into an analytical database and query that. So the question I started with was: what's the fewest components I co…

Just wanted to say thank you! Very relevant to our use cases. I'll report if I find any issues.

Welcome. Would love to hear your experience. Feel free to share here or in the repo. Fully open source.

Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire

#27

Post by vira28 »

raducu wrote: post #21
vira28 wrote: post #5

Author here. For context, I was the tech lead for the Postgres team at Cloudflare, and this came directly out of a challenge I kept hitting there: BI and dashboard teams needed to run long-running analytical queries, and the answer was always to spin up another bespoke read replica or stand up an ETL dump into an analytical database and query that. So the question I started with was: what's the fewest components I co…

> queryable from psql via an embedded DuckDB. noob question here from someone who ony played a bit with iceberg and trino: what's the reason to do the analytics stil inside the postgres -- is it so that you don't eat up the IOPS/bandwidth of the main postgresql disks?

[flagged]

Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire

#28

Post by vira28 »

erikcw wrote: post #19
vira28 wrote: post #5

Author here. For context, I was the tech lead for the Postgres team at Cloudflare, and this came directly out of a challenge I kept hitting there: BI and dashboard teams needed to run long-running analytical queries, and the answer was always to spin up another bespoke read replica or stand up an ETL dump into an analytical database and query that. So the question I started with was: what's the fewest components I co…

Thanks for releasing this! How do you handle DDL queries? Are table changes synchronized to the Iceberg table automatically? Also, I recently started looking into olake[0] to serve the same purpose. What would you say differentiates Streambed? [0] https://github.com/datazip-inc/olake

Thanks for the kind words!

Short answer: yes, column-level schema changes sync to Iceberg automatically[0].

Logical replication (pgoutput in v1) doesn't actually stream DDL statements. Instead, Postgres emits a fresh Relation message describing the table's current column layout right before the next change to that table. So we diff that against the last layout we knew and infer what changed.

From there we evolve the Iceberg schema in place: flush any buffered rows under the old schema first, then write a new metadata version with the change. What's handled today:

  - ADD COLUMN — new field ID allocated; the column's Postgres DEFAULT is carried into Iceberg's initial-default/write-default, so existing rows read back correctly
  - DROP COLUMN — removed from the current schema, existing data files untouched
  - Type widening — int4→int8, float4→float8 (the changes Iceberg considers compatible)
  - REPLICA IDENTITY changes
[0] https://github.com/viggy28/streambed/pull/21

Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire

#29

Post by vira28 »

viveknathani_ wrote: post #17

interesting approach, was exploring a Postgres to Clickhouse CDC setup while helping a team sometime back, this seems better as it allows separating the compute (query server) and storage (s3) layers, and thereby allowing us to be creative in cost reductions

Aside from the cost, my major motivation is to keep the infrastructure simple. The data is already there in Postgres, so I didn't want to add another data warehouse. I have also shared my thoughts on where this is heading https://viggy28.dev/article/postgres-gateway-drug/

Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire

#30

Post by kshri24 »

vira28 wrote: post #5

Author here. For context, I was the tech lead for the Postgres team at Cloudflare, and this came directly out of a challenge I kept hitting there: BI and dashboard teams needed to run long-running analytical queries, and the answer was always to spin up another bespoke read replica or stand up an ETL dump into an analytical database and query that. So the question I started with was: what's the fewest components I co…

> streams WAL changes straight into Apache Iceberg on S3, queryable from psql via an embedded DuckDB

Why not use Ducklake instead of Apache Iceberg? Wouldn't that simplify the architecture substantially?

Post reply on HN