Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Iceberg

iceberg.apache.org

Post reply on HN
Print view View original

31–40 of 68 posts

Re: Apache Iceberg

#31

Post by mritchie712 »

nikolatt wrote: post #22

I've been looking at Iceberg for a while, but in the end went with Delta Lake because it doesn't have a dependency on a catalog. It also has good support for reading and writing from it without needing Spark. Does anyone know if Iceberg has plans to support similar use cases?

Why don't you want a catalog? The SQL or REST catalogs are pretty light to set up. I have my eye on lakekeeper[0], but Polaris (from Snowflake) is a good option too.

PyIceberg is likely the easiest way to write without Spark.

0 - https://github.com/lakekeeper/lakekeeper

Re: Apache Iceberg

#32

Post by mritchie712 »

volderette wrote: post #15

How do you query your iceberg tables? We are looking into moving away from Bigquery and Starrocks [1] looks like a good option. [1] https://www.starrocks.io/

right now, starrocks or trino are likely your best options, but all the major query engines (clickhouse, snowflake, databricks, even duckdb) are improving their support too.

Re: Apache Iceberg

#33

Post by mritchie712 »

benjaminwootton wrote: post #20
varsketiz wrote: post #3

I'm somewhat surprised to see it here - Iceberg is around for some time already.

It’s been on the up in recent years though as it appears to have won the format wars. Every vendor is rallying around it and there were new open source catalogues and support from AWS at the end of 2024.

yeah, I'll admit I was worried when Databricks acquired Tabular[0] that it would hurt Iceberg's momentum (e.g. databricks would push delta instead), but it seems the opposite has happened.

0 - https://www.definite.app/blog/databricks-tabular-acquisition

Re: Apache Iceberg

#34

Post by mritchie712 »

tlarkworthy wrote: post #26
pradeepchhetri wrote: post #18

ClickHouse has a solid Iceberg integration. It has an Iceberg table function[0] and Iceberg table engine[1] for interacting with Iceberg data stored in s3, gcs, azure, hadoop etc. [0] https://clickhouse.com/docs/en/sql-reference/table-functions... [1] https://clickhouse.com/docs/en/engines/table-engines/integra...

I would say it doesn't but it is actively working on it https://github.com/ClickHouse/ClickHouse/issues/52054

duckdb has the same issue[0], I submitted a PR, but it's been stalled

0 - https://github.com/duckdb/duckdb-iceberg/pull/78

Re: Apache Iceberg

#35

Post by benesch »

jl6 wrote: post #24
teleforce wrote: post #12

Apache Iceberg is one of the emerging Open Table Formats in addition to Delta Lake and Apache Hudi [1]. [1] Open Table Formats: https://www.starburst.io/data-glossary/open-table-formats/

The table on that page makes it look like all three of these are very similar, with schema evolution and partition evolution being the key differences. Is that really it? I’d also love to see a good comparison between “regular” Iceberg and AWS’s new S3 Tables.

Yes, the three major open table formats are all quite similar.

When AWS launched S3 Tables last month I wrote a blog post with my first impressions: https://meltware.com/2024/12/04/s3-tables

There may be more in depth comparisons available by now but it’s at least a good starting point for understanding how S3 Tables integrates with Iceberg.

Re: Apache Iceberg

#36

Post by dm03514 »

I think iceberg solves a lot of big data problems, for handling huge amounts of data on blob storage, including partitioning, compaction and ACID semantics.

I really like the way the catalog standard can decouple underlying storage as well.

My biggest concern is how inaccessible the implementations are, Java / spark has the only mature implementation right now,

Even DuckDB doesn’t support writing yet.

I built out a tool to stream data to iceberg which uses the python iceberg client:

https://www.linkedin.com/pulse/streaming-iceberg-using-sqlfl...

Re: Apache Iceberg

#37

Post by peschu »

enether wrote: post #27
rubenvanwyk wrote: post #11

And yet there's still no straightforward way to write directly to Iceberg tables from Javascript as far as I know.

for some reason it's really cumbersome to access this tech

I agree, as a long time Business Intelligence developer I‘m still confused and astounded with all the tooling and bits and pieces seemingly necessary to create analytics/dashboards with open source tools.

For years I used a proprietary solution like Qlik Sense for the whole journey from data extraction to a finished dashboard (mostly on-prem). Going from raw data to a finished dashboard is a matter of days (not weeks/month) with one single tool (and maybe some scripts for supporting tasks). There is some „scripting“ involved for loading and transforming data, but if you already understand data models (and maybe have some sql experience) it is very easy. The Dashboard creation itself does not need any coding at all.just drag and drop and some formulas like sum(amount).

But this a standalone tool and it is hard to integrate it into your own piece of software. From my experience, software developers have a much more complicated view on data handling. Often this is just the complexity of their use cases, sometimes it is just a lack of knowledge of data preparation for analytics use cases.

Another part which complicates stuff greatly is the focus on use-cases involving cloud storage and doing all the transformations on distributed systems.

And it is often not clear what amount of data we are talking about and if it is realtime (streaming) data or not. There is a big difference in the possible approaches, if you have 6h hours to prepare data or if it has to be refreshed every second (or when new data arrives etc).

Long story short: Yes it is complicated to grasp. There is also a big difference if you use the data for normal analytics use cases in a company (mostly read only data models) or if you use the data in a (big tech) product.

I would suggest to start simple by looking into a „query engine“ to extract some data from somewhere and then doing some transformations with pandas/polars/cubejs for basic understanding. You will need some schedulers and orchestration on the way forward. But this will be dependent on the real use cases and environment you are in.

Re: Apache Iceberg

#38

Post by jl6 »

benesch wrote: post #35
jl6 wrote: post #24

Earlier quoted context omitted.

The table on that page makes it look like all three of these are very similar, with schema evolution and partition evolution being the key differences. Is that really it? I’d also love to see a good comparison between “regular” Iceberg and AWS’s new S3 Tables.

Yes, the three major open table formats are all quite similar. When AWS launched S3 Tables last month I wrote a blog post with my first impressions: https://meltware.com/2024/12/04/s3-tables There may be more in depth comparisons available by now but it’s at least a good starting point for understanding how S3 Tables integrates with Iceberg.

Cool, thank you. It feels like Athena + S3 Tables has the potential to be a very attractive serverless data lakehouse combo.

Re: Apache Iceberg

#39

Post by Icathian »

teleforce wrote: post #12

Apache Iceberg is one of the emerging Open Table Formats in addition to Delta Lake and Apache Hudi [1]. [1] Open Table Formats: https://www.starburst.io/data-glossary/open-table-formats/

I think this mischaracterizes the state of the space. Iceberg is the winner of this competition, as of a few months ago. All major vendors who didn't directly invent one of the others now support iceberg or have announced plans to do so.

Building lakehouse products on any table format but iceberg starting now seems to me like it must be a mistake.

Re: Apache Iceberg

#40

Post by bdndndndbve »

Icathian wrote: post #39
teleforce wrote: post #12

Apache Iceberg is one of the emerging Open Table Formats in addition to Delta Lake and Apache Hudi [1]. [1] Open Table Formats: https://www.starburst.io/data-glossary/open-table-formats/

I think this mischaracterizes the state of the space. Iceberg is the winner of this competition, as of a few months ago. All major vendors who didn't directly invent one of the others now support iceberg or have announced plans to do so. Building lakehouse products on any table format but iceberg starting now seems to me like it must be a mistake.

Yeah working in the data space I see a ton of customers using Iceberg and some using Delta Lake if they're already a Databricks shop. Virtually no Hudi.
Post reply on HN