Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire
1–10 of 47 posts
Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire
#2Replicating the Postgres WAL to S3 and Iceberg reliably is a hard problem but it’s not accurate to say that no ETL is needed here.
maybe you can say it’s more of an ELT pattern but anyone who’s interested into using this for realistic analytics they will have to transform the data at some point.
If an org is early enough to think that they can use a solution like this and just get in duckdb and start spitting out reports, they will be up for a really bad experience.
Please educate people to do the right thing and realize the scope of the work they are facing, it might feel that it hurts your growth in the short term but it will benefit you greatly in the mid-long term as a vendor.
Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire
#3Replicating the Postgres WAL to S3 and Iceberg reliably is a hard problem but it’s not accurate to say that no ETL is needed here. maybe you can say it’s more of an ELT pattern but anyone who’s interested into using this for realistic analytics they will have to transform the data at some point. If an org is early enough to think that they can use a solution like this and just get in duckdb and start spitting out rep…
IDK, AWS Zero ETL from Autora into Redshift really helped us at some point. You right that data transformation is very limited if not possible. But having data in an analytical store, being able to experiment with queries, understand what is wrong with your OLTP schema and then build ETL is way better than doing an upfront design.
Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire
#4Hi, this looks interesting, thanks for sharing. I am the builder of ingestr (https://github.com/bruin-data/ingestr), so I am very much in the same space.
I really like that you did this in Go, and I'll definitely dig a bit more into the source code to see how you tackled the CDC stuff, given that there is not many reliable CDC libraries in Go, and there are quite a few gotchas when it comes to doing CDC right. We also hand-rolled ours in ingestr, or I must say clanker-rolled, and we got quite a few things wrong in the first place.
Curious about the postgres-compatible query option: what's the usecase you have in mind there? My perception is that any org that would use Iceberg also has one or a few query engines in place, is this more for debugging stuff?
Quite cool stuff, keep it up!
Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire
#5Author here. For context, I was the tech lead for the Postgres team at Cloudflare, and this came directly out of a challenge I kept hitting there: BI and dashboard teams needed to run long-running analytical queries, and the answer was always to spin up another bespoke read replica or stand up an ETL dump into an analytical database and query that.
So the question I started with was: what's the fewest components I could get away with? That led to the architecture here — Streambed connects to Postgres as a logical replication subscriber (same mechanism as a read replica) and streams WAL changes straight into Apache Iceberg on S3, queryable from psql via an embedded DuckDB. There are a lot of edge cases to handle, and it's very much early days.
Welcome any feedback.
Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire
#6Replicating the Postgres WAL to S3 and Iceberg reliably is a hard problem but it’s not accurate to say that no ETL is needed here. maybe you can say it’s more of an ELT pattern but anyone who’s interested into using this for realistic analytics they will have to transform the data at some point. If an org is early enough to think that they can use a solution like this and just get in duckdb and start spitting out rep…
[flagged]
Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire
#7Hi, this looks interesting, thanks for sharing. I am the builder of ingestr ( https://github.com/bruin-data/ingestr ), so I am very much in the same space. I really like that you did this in Go, and I'll definitely dig a bit more into the source code to see how you tackled the CDC stuff, given that there is not many reliable CDC libraries in Go, and there are quite a few gotchas when it comes to doing CDC right. We a…
Hello, I checked ingestr repo, and it is in my bookmark. Small world.
Agree, CDC is like Death by a thousand cuts. I believe Debezium has a Java library.
My initial need was Postgres compatibilty. Wanted to give an endpoint that BI and dashboard teams can use to query as if they are querying a Postgres replica. Added more context here https://news.ycombinator.com/item?id=48350820
Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire
#8Author here. For context, I was the tech lead for the Postgres team at Cloudflare, and this came directly out of a challenge I kept hitting there: BI and dashboard teams needed to run long-running analytical queries, and the answer was always to spin up another bespoke read replica or stand up an ETL dump into an analytical database and query that. So the question I started with was: what's the fewest components I co…
Just wanted to say thank you! Very relevant to our use cases. I'll report if I find any issues.
Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire
#9nice work! we have handrolled
something similar at work.
do you have any perf metrics? throughput, end-to-end latency, etc?
Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire
#10Replicating the Postgres WAL to S3 and Iceberg reliably is a hard problem but it’s not accurate to say that no ETL is needed here. maybe you can say it’s more of an ELT pattern but anyone who’s interested into using this for realistic analytics they will have to transform the data at some point. If an org is early enough to think that they can use a solution like this and just get in duckdb and start spitting out rep…
IDK, AWS Zero ETL from Autora into Redshift really helped us at some point. You right that data transformation is very limited if not possible. But having data in an analytical store, being able to experiment with queries, understand what is wrong with your OLTP schema and then build ETL is way better than doing an upfront design.
Of course it is. What you describe is one of the reasons that ELT became popular, if you couple it with a variant type and schema on read, you have a very powerful and flexible architecture.
But there’s no free lunch, building and maintains data infrastructure that is reliable requires work. Many companies don’t realise that when they start their analytical journey and aggressive marketing doesn’t help. That’s the point I was trying to make.