Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Iceberg

iceberg.apache.org

Post reply on HN
Print view View original

51–60 of 68 posts

Re: Apache Iceberg

#51

Post by volderette »

jl6 wrote: post #23
volderette wrote: post #15

How do you query your iceberg tables? We are looking into moving away from Bigquery and Starrocks [1] looks like a good option. [1] https://www.starrocks.io/

Why away from bigquery? Just wondering if it’s a cost thing.

Yes, mainly driven by cost. BigQuery is really unpredictable when dashboards with filters are being used intensively by users. We don’t want to limit our users in their data exploration.

Re: Apache Iceberg

#52

Post by npalli »

Are there robust non-JVM based implementations for Iceberg currently? Sorry to say, but recommending JVM ecosystems around large data just feels like professional malpractice at this point. Whether deployment complexity, resource overhead, tool sprawl or operational complexity the ecosystem seems to attract people who solve only 50% of the problem and have another tool to solve the rest, which in turn only solves 50% etc.. ad infinitum. The popularity of solutions like Snowflake, Clickhouse, or DuckDB is not an accident and is the direction everything should go. I hear Snowflake will adopt this in the future, that is good news.

Re: Apache Iceberg

#53

Post by juunpp »

npalli wrote: post #52

Are there robust non-JVM based implementations for Iceberg currently? Sorry to say, but recommending JVM ecosystems around large data just feels like professional malpractice at this point. Whether deployment complexity, resource overhead, tool sprawl or operational complexity the ecosystem seems to attract people who solve only 50% of the problem and have another tool to solve the rest, which in turn only solves 50%…

> who solve only 50% of the problem and have another tool to solve the rest, which in turn only solves 50% etc.. ad infinitum

This actually converges to 1:

1/2 + 1/4 + 1/8 + 1/16 + ... = 1

You just need 30kloc of maven in your pom before you get there.

Re: Apache Iceberg

#54

Post by romperstomper »

mritchie712 wrote: post #30

If you're looking to give Iceberg a spin, here's how to get it running locally, on AWS[0] and on GCP[1]. The posts use DuckDB as the query engine, but you could swap in Trino (or even chdb / clickhouse). 0 - https://www.definite.app/blog/cloud-iceberg-duckdb-aws 1 - https://www.definite.app/blog/cloud-iceberg-duckdb

you can just use iceberg tables with AWS Glue/Athena

Re: Apache Iceberg

#57

Post by chehai »

In order to get good query performance from Iceberg, we have to run compaction frequently. Compaction turns out to be very expensive. Any tip to minimize compaction while keeping queries fast?

Re: Apache Iceberg

#58

Post by hipadev23 »

romperstomper wrote: post #54
mritchie712 wrote: post #30

If you're looking to give Iceberg a spin, here's how to get it running locally, on AWS[0] and on GCP[1]. The posts use DuckDB as the query engine, but you could swap in Trino (or even chdb / clickhouse). 0 - https://www.definite.app/blog/cloud-iceberg-duckdb-aws 1 - https://www.definite.app/blog/cloud-iceberg-duckdb

you can just use iceberg tables with AWS Glue/Athena

aws glue/athena has the most absurd setup process. duckdb and clickhouse is “select * from s3(…)”

Re: Apache Iceberg

#59

Post by czwief »

volderette wrote: post #15

How do you query your iceberg tables? We are looking into moving away from Bigquery and Starrocks [1] looks like a good option. [1] https://www.starrocks.io/

Starburst (full disclosure: I work there) provides a query engine (trino under the hood) with Iceberg support [1] -- worth checking out.

[1] https://www.starburst.io/platform/icehouse/

Re: Apache Iceberg

#60

Post by tlarkworthy »

tlarkworthy wrote: post #26
pradeepchhetri wrote: post #18

ClickHouse has a solid Iceberg integration. It has an Iceberg table function[0] and Iceberg table engine[1] for interacting with Iceberg data stored in s3, gcs, azure, hadoop etc. [0] https://clickhouse.com/docs/en/sql-reference/table-functions... [1] https://clickhouse.com/docs/en/engines/table-engines/integra...

I would say it doesn't but it is actively working on it https://github.com/ClickHouse/ClickHouse/issues/52054

Oh they just fixed this 9d ago and I guess this comment provoked them to close the issue!
Post reply on HN