I really liked your article. Is this a typo: “Hive, Delta Lake and Iceberg all support support of schema registry or metastore.”?
Thanks for the typo, fixed!
Understanding Parquet, Iceberg and Data Lakehouses
101–107 of 107 posts
Re: Understanding Parquet, Iceberg and Data Lakehouses
#102I am very excited about Iceberg specifically (because open-source), but the last time I looked into it the only implementation was a Spark library, and Trino's (formerly Presto, an SQL engine) Iceberg connector had a hard dependency on Hive! It is like the entire industry had a hard time divorcing its MapReduce, Hive, and dare I to say Spark, legacy. I didn't look into Iceberg since, but plan to, and I am really look…
Trino no longer depends on Hadoop/Hive for any of its data lake connectors. Removing that dependency was a huge effort.
Re: Understanding Parquet, Iceberg and Data Lakehouses
#103I often hear references to Apache Iceberg and Delta Lake as if they’re two peas in the Open Table Formats pod. Yet… Here’s the Apache Iceberg table format specification: https://iceberg.apache.org/spec/ As they like to say in patent law, anyone “skilled in the art” of database systems could use this to build and query Iceberg tables without too much difficulty. This is nominally the Delta Lake equivalent: https://git…
If you are a Spark shop then choosing Delta over Iceberg is a no-brainer. It's simpler and perfectly integrated. Not to mention that the Spark's Delta connector can now generate Iceberg-compatible metadata too. The choice between the two resembles the choice between Parquet and ORC circa 2016. Two formats of broadly the same power, initially biased by a particular query engine, eventually at feature parity and univer…
I feel like this is frankly uninformed. Many iceberg shops seem to rely heavily on Spark as a primary engine. And databricks has a history of being a hostile oss force with the culture of the spark project being toxic from the start and delta’s questionable commitment to being a community project.
Re: Understanding Parquet, Iceberg and Data Lakehouses
#104I've heard of data lakes, but "data lakehouse" sounds like where upper class data goes in the summer to take their data-boats data-fishing.
I never understood what is meant by “data lake” in the first place, other than “heterogenous collection of large-ish data files”.
During the big data hype, I did a feasibility study for this for my previous company (typical bigcorp) and learned that most companies should not have a data lake.
A data lake is a collection of different types of structured/unstructured data like CSV, Parquet, text, images, etc. stored in an object store or some such that in principle you're able to query. The theory is that you can just dump stuff into a kitchen drawer (ELT instead of ETL) and be able to do analytics on it later.
But most enterprises already have huge investments in relational databases (SQL Server, Oracle etc.) which are decades-old optimized, typed with schema, structured engines for storing data. If you have a SQL database, chances are you already have data in the right format for analytics and building a data lake is the wrong way to go.
People in tech companies have this wrong impression that enterprises have a lot of big data, but the fact is, most of the valuable data in most companies are less than a few terabytes total. They're mostly ERP data, Excel files, and operational data from various sensors (if that).
To unstructure the (already structured) data just so it can fit into the data lake seemed like the wrong strategy, but I was surprised how much companies like Cloudera and others hyped it up so much so they could sell technologies like Hive, Spark, Presto, etc. (and streaming tech like Kafka). These are overkill for most enterprises.
Re: Understanding Parquet, Iceberg and Data Lakehouses
#105No mention of Hudi? I really liked using Hudi in a recent project. It feels so close to hitting that maturity level where it’s viable for a small team to maintain without introducing too many living parts.
Overall, I like the whole concept of the Lakehouse because it can be done cheaply.
Most datalakes turn into swamps pretty quickly, so cheaper is better.
Let it sit unused for a while in S3 and then quietly nuke it without burning money on a big compute environment.
Re: Understanding Parquet, Iceberg and Data Lakehouses
#106I often hear references to Apache Iceberg and Delta Lake as if they’re two peas in the Open Table Formats pod. Yet… Here’s the Apache Iceberg table format specification: https://iceberg.apache.org/spec/ As they like to say in patent law, anyone “skilled in the art” of database systems could use this to build and query Iceberg tables without too much difficulty. This is nominally the Delta Lake equivalent: https://git…
The fact that they use JSON for delta changes is... just stupid. For contrast, in SQL Server, it's implemented way better. Columnar storage tables (columnstore indexes, an equivalent of Parquet or ORC inside the engine) are immutable, and deltas are stored in B-Trees for compactness, ease of access and speed). At some point in time the columnstores get defragmented/merged/rebuilt in part or in whole, and the B-Tree is deleted and starts over when new changes accumulate. Doing it in JSON is, let me put it softly, a sign of bad times.
I suppose anything is better than Delta Lake. Especially Iceberg.
Re: Understanding Parquet, Iceberg and Data Lakehouses
#107One thing I'm confused about is why does Iceberg need a spark deployment to function? Or am I wrong about that? I would rather avoid that ecosystem if I can.
You don't need a Spark deployment. The first reference implementations for reading and writing were in Spark. Now, with PyIceberg, there is read support in Python. Write support should be merged very soon - https://github.com/apache/iceberg-python/pull/41 So, very soon, you will be able to read/write Iceberg tables in Python. I look forward to doing data transformations in Polars for data of reasonable scale (up to 1…
Well, what about other languages? Every language needs bindings or a re-implementation? (i.e., iceberg tables are written/queried in-process as opposed to via a network API?)