Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding Parquet, Iceberg and Data Lakehouses

davidgomes.com

Post reply on HN
Print view View original

31–40 of 107 posts

Re: Understanding Parquet, Iceberg and Data Lakehouses

#31

Post by hacful-tonteg »

jamesblonde wrote: post #9
twoodfin wrote: post #5

I often hear references to Apache Iceberg and Delta Lake as if they’re two peas in the Open Table Formats pod. Yet… Here’s the Apache Iceberg table format specification: https://iceberg.apache.org/spec/ As they like to say in patent law, anyone “skilled in the art” of database systems could use this to build and query Iceberg tables without too much difficulty. This is nominally the Delta Lake equivalent: https://git…

I agree with all of this. Databricks are also holding back features from open source Delta (like bloom filters), which is their right. But then you can't claim it is a community-driven open format, unless it is an animal farm version of that, where one of the versions is the Pig (some are more equal than others).

Databricks has a lot of nice closed-sourced components, e.g., Unity Catalog, Delta Live Tables and Photon (a C++ implementation of Spark).

Delta itself seems fairly open-source: https://github.com/orgs/delta-io/projects/10/views/1 and hopefully someone will implement Liquid Clustering!

Re: Understanding Parquet, Iceberg and Data Lakehouses

#32

Post by hacful-tonteg »

twoodfin wrote: post #5

I often hear references to Apache Iceberg and Delta Lake as if they’re two peas in the Open Table Formats pod. Yet… Here’s the Apache Iceberg table format specification: https://iceberg.apache.org/spec/ As they like to say in patent law, anyone “skilled in the art” of database systems could use this to build and query Iceberg tables without too much difficulty. This is nominally the Delta Lake equivalent: https://git…

Microsoft is using Delta for their Fabric Lakehouse architecture and its also what OneLake is built around so now you have another massive player choosing Delta.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#33

Post by krysp »

wokwokwok wrote: post #25

It's really easy to get lost in the technical jargon that the vendors who are selling products throw around, but this article has missed the important part, and spent all the time talking about the relatively unimportant part (data formats). You need to step back and look from a broader perspective to understand this domain. Talking about arrow/parquet/iceberg is like talking about InnoDB vs MyISAM when you're talkin…

All very good and useful points. One additional thing to mention is that as you are querying across the raw data with a data lake(house), performance is fundamentally worse, even if a lot of the marketing material will tell you otherwise. Usually significantly worse than if your data was in a columnar database in practice.

Depending on your use case this may or may not be a problem. For most companies I'd wager that it is a bigger problem than it first appears.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#34

Post by jamesblonde »

wokwokwok wrote: post #25

It's really easy to get lost in the technical jargon that the vendors who are selling products throw around, but this article has missed the important part, and spent all the time talking about the relatively unimportant part (data formats). You need to step back and look from a broader perspective to understand this domain. Talking about arrow/parquet/iceberg is like talking about InnoDB vs MyISAM when you're talkin…

All of this is correct, but it misses the main point of the new table formats - they are open-source and the data can be stored on very low cost storgae - S3. So, having a data warehouse that stores TBs or even PBs of data is not as expensive as it used to be (by an order of magnitude or more). And the formats for storing the data (Parquet), its metadata (Iceberg, Hudi, delta lake), its query engine (DuckDB, Polars, Ibis) - they are all open-source.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#35

Post by 3abiton »

wenc wrote: post #2

Great article. I've worked with Parquet files on S3 for years, but I didn't quite understand what Iceberg was, but the article explained it well. It's a database metadata format for an underlying set of data which describes its schema, partitioning etc. Most people use Hive partitioning convention (i.e. directory names like /key3=000/key2=002/) but Iceberg goes farther than this by exposing even more structure to the…

Very grateful of your recap, I skimmed through the article fast, but got a better understanding reading your comment!

Re: Understanding Parquet, Iceberg and Data Lakehouses

#36

Post by isbvhodnvemrwvn »

benjaminwootton wrote: post #16
debo_ wrote: post #6

I've heard of data lakes, but "data lakehouse" sounds like where upper class data goes in the summer to take their data-boats data-fishing.

The name is easy to poke fun at, but I think it’s a real problem. A lot of companies use data lakes to store data and warehouses to serve BI to tools like Tableau or PowerBI. They then up copying data between the two. Querying a lake directly and having transactions, governance etc against one set of data (a data Lakehouse) can really simplify the stack and take out cost.

Ah, so the house part comes from warehouse. Not obvious to say the least.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#38

Post by apwell23 »

benjaminwootton wrote: post #14

This is a big deal in the database world as delta, iceberg and hudi mean that data is being stored in an open source format, often on S3. It means that the storage and much of the processing is being standrdised so that you can move between databases easily and almost all tools will eventually be able to work with the same set of files in a transactionally sound way. For instance, Snowflake could be writing to a file…

> " every database vendor will be forced by the market to optimise for performance such that they tend towards the performance of natively ingested data."

This assumes that their internal storage format has nothing to do with decades of engineering infrastructure that they built their business model around and that they would simply give all that up and compete based on just their compute layer. snowflake might as well shutup shop and return billions to the investors. Locking in data into their ecosystem is their whole business model.

Is there as good example of open standard forcing companies to give up their proprietary tech ?

Re: Understanding Parquet, Iceberg and Data Lakehouses

#39

Post by apwell23 »

nerdponx wrote: post #27
snthpy wrote: post #8

Earlier quoted context omitted.

Thanks for this. I've been following this space for about a year or two and was wondering why Iceberg was more popular in open source. Over the past six months I got the impression that Delta is pulling ahead in the race as Iceberg is struggling to provide tools for people not in the JVM ecosystem. Delta is a lot more accessible in that way.

Snowflake is rolling out Iceberg support and not Delta support, I think that says a lot.

Bigquery too.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#40

Post by twoodfin »

hacful-tonteg wrote: post #32
twoodfin wrote: post #5

I often hear references to Apache Iceberg and Delta Lake as if they’re two peas in the Open Table Formats pod. Yet… Here’s the Apache Iceberg table format specification: https://iceberg.apache.org/spec/ As they like to say in patent law, anyone “skilled in the art” of database systems could use this to build and query Iceberg tables without too much difficulty. This is nominally the Delta Lake equivalent: https://git…

Microsoft is using Delta for their Fabric Lakehouse architecture and its also what OneLake is built around so now you have another massive player choosing Delta.

Yes, another massive player who has the resources and independent market pull to ride and steer a complex and ever-shifting “standard”.

Feels a bit like, “If Delta Lake did not exist, Microsoft would have to invent it.”

Post reply on HN