Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding Parquet, Iceberg and Data Lakehouses

davidgomes.com

Post reply on HN
Print view View original

1–10 of 107 posts

Re: Understanding Parquet, Iceberg and Data Lakehouses

#2

Post by wenc »

Great article. I've worked with Parquet files on S3 for years, but I didn't quite understand what Iceberg was, but the article explained it well. It's a database metadata format for an underlying set of data which describes its schema, partitioning etc.

Most people use Hive partitioning convention (i.e. directory names like /key3=000/key2=002/) but Iceberg goes farther than this by exposing even more structure to the query engine.

In a traditional DBMS like Postgres, the schema, the query engine and the storage format come as a single package.

But with big data, we're building database components from scratch, and we can mix and match. We can use Iceberg as a metadata format, DuckDB as the query engine, Parquet as the storage format, and S3 as the storage medium.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#3

Post by albert_e »

Sorry genuine question -- what does the phrase "at Broad" at the end of the blog post's title mean or refer to? Maybe a phrase that I am unfamiliar with? I first wondered if it is the name of an organization or team -- and this post is describing what they did in that team, but that doesn't seem to be the case?

>> Understanding Parquet, Iceberg and Data Lakehouses at Broad

Re: Understanding Parquet, Iceberg and Data Lakehouses

#4

Post by chomp5977 »

albert_e wrote: post #3

Sorry genuine question -- what does the phrase "at Broad" at the end of the blog post's title mean or refer to? Maybe a phrase that I am unfamiliar with? I first wondered if it is the name of an organization or team -- and this post is describing what they did in that team, but that doesn't seem to be the case? >> Understanding Parquet, Iceberg and Data Lakehouses at Broad

Not in depth, general understanding

Re: Understanding Parquet, Iceberg and Data Lakehouses

#5

Post by twoodfin »

I often hear references to Apache Iceberg and Delta Lake as if they’re two peas in the Open Table Formats pod. Yet…

Here’s the Apache Iceberg table format specification:

https://iceberg.apache.org/spec/

As they like to say in patent law, anyone “skilled in the art” of database systems could use this to build and query Iceberg tables without too much difficulty.

This is nominally the Delta Lake equivalent:

https://github.com/delta-io/delta/blob/master/PROTOCOL.md

I defy anyone to even scope out what level of effort would be required to fully implement the current spec, let alone what would be involved in keeping up to date as this beast evolves.

Frankly, the Delta Lake spec reads like a reverse engineering of whatever implementation tradeoffs Databricks is making as they race to build out a lakehouse for every Fortune 1000 company burned by Hadoop (which is to say, most of them).

My point is that I’ve yet to be convinced that buying into Delta Lake is actually buying into an open ecosystem. Would appreciate any reassurance on this front!

Editing to append this GitHub history, which is unfortunately not reassuring:

https://github.com/delta-io/delta/commits/master/PROTOCOL.md

Random features and tweaks just popping up, PR’d by Databricks engineers and promptly approved by Databricks senior engineers…

Re: Understanding Parquet, Iceberg and Data Lakehouses

#8

Post by snthpy »

twoodfin wrote: post #5

I often hear references to Apache Iceberg and Delta Lake as if they’re two peas in the Open Table Formats pod. Yet… Here’s the Apache Iceberg table format specification: https://iceberg.apache.org/spec/ As they like to say in patent law, anyone “skilled in the art” of database systems could use this to build and query Iceberg tables without too much difficulty. This is nominally the Delta Lake equivalent: https://git…

Thanks for this. I've been following this space for about a year or two and was wondering why Iceberg was more popular in open source.

Over the past six months I got the impression that Delta is pulling ahead in the race as Iceberg is struggling to provide tools for people not in the JVM ecosystem. Delta is a lot more accessible in that way.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#9

Post by jamesblonde »

twoodfin wrote: post #5

I often hear references to Apache Iceberg and Delta Lake as if they’re two peas in the Open Table Formats pod. Yet… Here’s the Apache Iceberg table format specification: https://iceberg.apache.org/spec/ As they like to say in patent law, anyone “skilled in the art” of database systems could use this to build and query Iceberg tables without too much difficulty. This is nominally the Delta Lake equivalent: https://git…

I agree with all of this. Databricks are also holding back features from open source Delta (like bloom filters), which is their right. But then you can't claim it is a community-driven open format, unless it is an animal farm version of that, where one of the versions is the Pig (some are more equal than others).

Re: Understanding Parquet, Iceberg and Data Lakehouses

#10

Post by jamesblonde »

snthpy wrote: post #8
twoodfin wrote: post #5

I often hear references to Apache Iceberg and Delta Lake as if they’re two peas in the Open Table Formats pod. Yet… Here’s the Apache Iceberg table format specification: https://iceberg.apache.org/spec/ As they like to say in patent law, anyone “skilled in the art” of database systems could use this to build and query Iceberg tables without too much difficulty. This is nominally the Delta Lake equivalent: https://git…

Thanks for this. I've been following this space for about a year or two and was wondering why Iceberg was more popular in open source. Over the past six months I got the impression that Delta is pulling ahead in the race as Iceberg is struggling to provide tools for people not in the JVM ecosystem. Delta is a lot more accessible in that way.

I guess you are referring to delta-rs (for Python in particular). An interesting factoid here is that Databricks started delta-rs, and other companies are now driving it forward - not Databricks. I guess it is not in Databricks interest to push the non JVM ecosystem. PyIceberg is catching up. Write support is almost there - https://github.com/apache/iceberg-python/pull/41
Post reply on HN