Apache Iceberg • Hacker News Board

Re: Apache Iceberg

#61

Post by pradeepchhetri » Mon, Jan 27, 2025, 7:00 AM UTC

tlarkworthy wrote: post #60

tlarkworthy wrote: post #26

Earlier quoted context omitted.

I would say it doesn't but it is actively working on it https://github.com/ClickHouse/ClickHouse/issues/52054

Oh they just fixed this 9d ago and I guess this comment provoked them to close the issue!

I am looking forward to learn about such upcoming features in the community call https://clickhouse.com/company/events/v25-1-community-releas...

Re: Apache Iceberg

#62

Post by datancoffee » Mon, Jan 27, 2025, 8:51 AM UTC

mritchie712 wrote: post #31

nikolatt wrote: post #22

I've been looking at Iceberg for a while, but in the end went with Delta Lake because it doesn't have a dependency on a catalog. It also has good support for reading and writing from it without needing Spark. Does anyone know if Iceberg has plans to support similar use cases?

Why don't you want a catalog? The SQL or REST catalogs are pretty light to set up. I have my eye on lakekeeper[0], but Polaris (from Snowflake) is a good option too. PyIceberg is likely the easiest way to write without Spark. 0 - https://github.com/lakekeeper/lakekeeper

We did an evaluation of various REST catalog options and went with Open Catalog from Snowflake (a Polaris-based managed service that works independently from their data warehousing solution). Lakekeeper is nice - it's one of the few catalogs with FGAC and table maintenance.

https://tower.dev/blog/picking-snowflake-open-catalog-as-a-m...

Re: Apache Iceberg

#63

Post by romperstomper » Mon, Jan 27, 2025, 8:14 PM UTC

hipadev23 wrote: post #58

romperstomper wrote: post #54

Earlier quoted context omitted.

you can just use iceberg tables with AWS Glue/Athena

aws glue/athena has the most absurd setup process. duckdb and clickhouse is “select * from s3(…)”

I'm not sure what do you mean by "absurd setup". In case of Athena you just use the iceberg type for a table as you create it and that's it. Under the hood AWS also uses Trino or Presto as far as I know.

Re: Apache Iceberg

#64

Post by adesh_nalpet » Mon, Jan 27, 2025, 9:23 PM UTC

mritchie712 wrote: post #30

If you're looking to give Iceberg a spin, here's how to get it running locally, on AWS[0] and on GCP[1]. The posts use DuckDB as the query engine, but you could swap in Trino (or even chdb / clickhouse). 0 - https://www.definite.app/blog/cloud-iceberg-duckdb-aws 1 - https://www.definite.app/blog/cloud-iceberg-duckdb

You can just used Iceberg Java API: https://iceberg.apache.org/docs/1.6.1/api/#file-level

Use it with Dropwizard/Springboot, you get to expose rest APIs too.

Re: Apache Iceberg

#65

Post by adesh_nalpet » Mon, Jan 27, 2025, 9:29 PM UTC

volderette wrote: post #15

How do you query your iceberg tables? We are looking into moving away from Bigquery and Starrocks [1] looks like a good option. [1] https://www.starrocks.io/

Common opensource options (other than Spark and Flink): 1. Dremio: https://www.dremio.com/ 2. Trino: https://trino.io/ 3. Iceberg Java API: https://iceberg.apache.org/docs/1.6.1/api/

Re: Apache Iceberg

#66

Post by jaakl » Tue, Jan 28, 2025, 6:31 AM UTC

apwell23 wrote: post #43

I am stockholder in snowflake and iceberg's ascendance seems to coincide with snow's downfall. Is the query engine value add justify snowflake's valuation. Their data marketplace thing didn't seem to have actually worked.

I’m doing datalake modernization for medium-large enterprise and spent last months in sales calls of MS Fabric vs Snowflake vs Databricks. All fun, but now with the managed Iceberg in AWS (S3 tables) I tend to consider to choose none of them: just plain Iceberg is good enough. Of course someone needs to write and read it; but there are so many good free options already, even build does not feel scary. So I would go to the short side in Snowflake in medium-long term (looking their current value prop at least). Databricks has maybe more future as it has ML/AI-first approach. In short term we might still start with SF (with its Iceberg features), as the alternative future stack needs to mature and establish a bit.

Re: Apache Iceberg

#67

Post by tlarkworthy » Tue, Jan 28, 2025, 8:57 AM UTC

pradeepchhetri wrote: post #61

tlarkworthy wrote: post #60

Earlier quoted context omitted.

Oh they just fixed this 9d ago and I guess this comment provoked them to close the issue!

I am looking forward to learn about such upcoming features in the community call https://clickhouse.com/company/events/v25-1-community-releas...

oh and now the developer reopened it because it is not actually fully complete, lol. Yep, Iceburg on Clickhouse is WIP. I am actively watching this because it is relevant for my company.

Re: Apache Iceberg

#68

Post by pradeepchhetri » Thu, Jan 30, 2025, 5:00 AM UTC

tlarkworthy wrote: post #67

pradeepchhetri wrote: post #61

Earlier quoted context omitted.

I am looking forward to learn about such upcoming features in the community call https://clickhouse.com/company/events/v25-1-community-releas...

oh and now the developer reopened it because it is not actually fully complete, lol. Yep, Iceburg on Clickhouse is WIP. I am actively watching this because it is relevant for my company.

glad you are using CH :)