Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire

github.com

Post reply on HN
Print view View original

11–20 of 47 posts

Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire

#11

Post by kikimora »

cpard wrote: post #10
kikimora wrote: post #3

Earlier quoted context omitted.

IDK, AWS Zero ETL from Autora into Redshift really helped us at some point. You right that data transformation is very limited if not possible. But having data in an analytical store, being able to experiment with queries, understand what is wrong with your OLTP schema and then build ETL is way better than doing an upfront design.

Of course it is. What you describe is one of the reasons that ELT became popular, if you couple it with a variant type and schema on read, you have a very powerful and flexible architecture. But there’s no free lunch, building and maintains data infrastructure that is reliable requires work. Many companies don’t realise that when they start their analytical journey and aggressive marketing doesn’t help. That’s the po…

I don’t disagree, just placing emphasis on a different aspect.

In an ideal world there is a tool that moves your schema into an analytical store “as is” with a single click. Then the same tool lets you add arbitrary transformations of the data. Surprisingly I have not come across such a tool. It is earthier “one click to move your data” or “any transformation you want” but only after a significant upfront investment :(

Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire

#12

Post by kikimora »

vira28 wrote: post #5

Author here. For context, I was the tech lead for the Postgres team at Cloudflare, and this came directly out of a challenge I kept hitting there: BI and dashboard teams needed to run long-running analytical queries, and the answer was always to spin up another bespoke read replica or stand up an ETL dump into an analytical database and query that. So the question I started with was: what's the fewest components I co…

To me being able to query over psql is secondary. I’m fine with any SQL. What is very important is being able to transform the data to better suite analytical queries. That is, define custom transformations, define how data sectioned and what indices available.

Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire

#13

Post by buremba »

Looks interesting! It reminds me of pg_lake, which we evaluated for our startup https://lobu.ai but it's missing a lot of pushdown capabilities which made OLAP queries expensive.

I also tried DuckLake but that required us to move away from PG-first approach. I was thinking of using Debezium to create Iceberg on S3 for our append-only PG tables and use DuckDB. I will try Streambed out as well!

Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire

#14

Post by cpard »

kikimora wrote: post #11
cpard wrote: post #10

Earlier quoted context omitted.

Of course it is. What you describe is one of the reasons that ELT became popular, if you couple it with a variant type and schema on read, you have a very powerful and flexible architecture. But there’s no free lunch, building and maintains data infrastructure that is reliable requires work. Many companies don’t realise that when they start their analytical journey and aggressive marketing doesn’t help. That’s the po…

I don’t disagree, just placing emphasis on a different aspect. In an ideal world there is a tool that moves your schema into an analytical store “as is” with a single click. Then the same tool lets you add arbitrary transformations of the data. Surprisingly I have not come across such a tool. It is earthier “one click to move your data” or “any transformation you want” but only after a significant upfront investment…

I think I didn’t articulate myself very well on my reply. I actually wanted to say that I agree with you and emphasise again the need for educating users for the complexity of these projects.

What you describe has been pitched by many different products for different parts of the data platform. Fivetran for example claims to do that for the extraction and loading part, good old Informatica was offering the ETL in a graphical interface etc.

The problem that many teams ended up having is the explosion of the tooling needed by data teams.

Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire

#16

Post by vira28 »

buremba wrote: post #13

Looks interesting! It reminds me of pg_lake, which we evaluated for our startup https://lobu.ai but it's missing a lot of pushdown capabilities which made OLAP queries expensive. I also tried DuckLake but that required us to move away from PG-first approach. I was thinking of using Debezium to create Iceberg on S3 for our append-only PG tables and use DuckDB. I will try Streambed out as well!

Both projects are relevant. Curious, what kinda pushdown capabilities that you were looking for?

Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire

#17

Post by viveknathani_ »

interesting approach, was exploring a Postgres to Clickhouse CDC setup while helping a team sometime back, this seems better as it allows separating the compute (query server) and storage (s3) layers, and thereby allowing us to be creative in cost reductions

Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire

#18

Post by nylonstrung »

buremba wrote: post #13

Looks interesting! It reminds me of pg_lake, which we evaluated for our startup https://lobu.ai but it's missing a lot of pushdown capabilities which made OLAP queries expensive. I also tried DuckLake but that required us to move away from PG-first approach. I was thinking of using Debezium to create Iceberg on S3 for our append-only PG tables and use DuckDB. I will try Streambed out as well!

Does pushdown require support at this part of the stack or can you just delegate to Datafusion as your query engine, which has very good pushdown

Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire

#19

Post by erikcw »

vira28 wrote: post #5

Author here. For context, I was the tech lead for the Postgres team at Cloudflare, and this came directly out of a challenge I kept hitting there: BI and dashboard teams needed to run long-running analytical queries, and the answer was always to spin up another bespoke read replica or stand up an ETL dump into an analytical database and query that. So the question I started with was: what's the fewest components I co…

Thanks for releasing this! How do you handle DDL queries? Are table changes synchronized to the Iceberg table automatically?

Also, I recently started looking into olake[0] to serve the same purpose. What would you say differentiates Streambed?

[0] https://github.com/datazip-inc/olake

Re: Show HN: Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire

#20

Post by saisrirampur »

viveknathani_ wrote: post #17

interesting approach, was exploring a Postgres to Clickhouse CDC setup while helping a team sometime back, this seems better as it allows separating the compute (query server) and storage (s3) layers, and thereby allowing us to be creative in cost reductions

It depends on the use case. For real-time, customer-facing analytics, ClickHouse’s MergeTree engine is a natural fit, so a Postgres → ClickHouse CDC setup with low latencies (single-digit seconds) is better.

Replication to Iceberg/S3 is better suited for offline analytics and data warehousing use cases. You can use the same ClickHouse engine to query layer Iceberg data in S3.

Post reply on HN