We ran into issues with CDC when tables had a lot of TOAST columns. The WAL records don't include the full values unless you set REPLICA IDENTITY FULL. Does Streambed handle that, or do you need the extra config?
Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire
41–47 of 47 posts
Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire
#42Author here. For context, I was the tech lead for the Postgres team at Cloudflare, and this came directly out of a challenge I kept hitting there: BI and dashboard teams needed to run long-running analytical queries, and the answer was always to spin up another bespoke read replica or stand up an ETL dump into an analytical database and query that. So the question I started with was: what's the fewest components I co…
Hey vira28, thanks a lot for your work. This is a very promising project because other alternative like supabase/etl, Kuvasz-streamer, Sequin all have some subtle issues. Few questions: 1) For a supabase project can we setup replication slot on replica instead of primary? https://sequinstream.com/docs/reference/databases#using-sequ... 2) For a planetscale cluster are the replication slots on primary or the follower n…
Welcome. To avoid primary running out of disk space, you can configure max_slot_wal_keep_size https://www.postgresql.org/docs/17/runtime-config-replicatio...
Since Supabase is vanilla Postgres, streambed should work with replica as the source.
reg, Planetscale, I haven't looked at their offerings yet.
Where do you host your DB currently? Happy to try out with that provider as the source.
Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire
#43We ran into issues with CDC when tables had a lot of TOAST columns. The WAL records don't include the full values unless you set REPLICA IDENTITY FULL. Does Streambed handle that, or do you need the extra config?
Currently, Strembed expects REPLICA IDENTITY FULL for getting the before and after value of TOAST column. Since we have the data in object storage, we could populate it without the need for REPLICA IDENTITY FULL. Created an issue https://github.com/viggy28/streambed/issues/25 to track this feature.
Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire
#44This is a nice project! we do some exporting of data from postgres to s3 and its a little flaky but does the job for now. Feel like this a good project to explore using
The challenge with any CDC is making it reliable. Curious, how are you exporting to S3? - Debezium or some service in AWS or home grown tool?
Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire
#45Author here. For context, I was the tech lead for the Postgres team at Cloudflare, and this came directly out of a challenge I kept hitting there: BI and dashboard teams needed to run long-running analytical queries, and the answer was always to spin up another bespoke read replica or stand up an ETL dump into an analytical database and query that. So the question I started with was: what's the fewest components I co…
> streams WAL changes straight into Apache Iceberg on S3, queryable from psql via an embedded DuckDB Why not use Ducklake instead of Apache Iceberg? Wouldn't that simplify the architecture substantially?
From what I understand Ducklake needs a dedicated metadata database and it also ties to DuckDB land wherease with Iceberg many engines can query directly.
Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire
#46Author here. For context, I was the tech lead for the Postgres team at Cloudflare, and this came directly out of a challenge I kept hitting there: BI and dashboard teams needed to run long-running analytical queries, and the answer was always to spin up another bespoke read replica or stand up an ETL dump into an analytical database and query that. So the question I started with was: what's the fewest components I co…
How does it compare to https://github.com/supabase/etl ?
The idea is pretty similar. As per their README, Iceberg support is deprecated.
Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire
#47Earlier quoted context omitted.
> streams WAL changes straight into Apache Iceberg on S3, queryable from psql via an embedded DuckDB Why not use Ducklake instead of Apache Iceberg? Wouldn't that simplify the architecture substantially?
From what I understand Ducklake needs a dedicated metadata database and it also ties to DuckDB land wherease with Iceberg many engines can query directly.
You are already using Postgres in your setup so why not just use Postgres itself as metadata database? It is a much better setup than using Iceberg [1].
> and it also ties to DuckDB land wherease with Iceberg many engines can query directly.
Ducklake can also be queried by many engines [2]. Though not as exhaustive as Iceberg.
[1]: https://www.youtube.com/watch?v=-PYLFx3FRfQ [2]: https://ducklake.select/docs/stable/#list-of-ducklake-client...