Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding Parquet, Iceberg and Data Lakehouses

davidgomes.com

Post reply on HN
Print view View original

11–20 of 107 posts

Re: Understanding Parquet, Iceberg and Data Lakehouses

#11

Post by wenc »

snthpy wrote: post #8
twoodfin wrote: post #5

I often hear references to Apache Iceberg and Delta Lake as if they’re two peas in the Open Table Formats pod. Yet… Here’s the Apache Iceberg table format specification: https://iceberg.apache.org/spec/ As they like to say in patent law, anyone “skilled in the art” of database systems could use this to build and query Iceberg tables without too much difficulty. This is nominally the Delta Lake equivalent: https://git…

Thanks for this. I've been following this space for about a year or two and was wondering why Iceberg was more popular in open source. Over the past six months I got the impression that Delta is pulling ahead in the race as Iceberg is struggling to provide tools for people not in the JVM ecosystem. Delta is a lot more accessible in that way.

DuckDB (lightweight, non-JVM, many language bindings) supports querying from Iceberg now.

https://duckdb.org/docs/extensions/iceberg.html

You still need Spark to generate the Iceberg metadata though.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#12

Post by jamesblonde »

I disagree with this strongly - "The best way to store Apache Arrow dataframes in files on disk is with Feather. However, it’s also possible to convert to Apache Parquet format and others."

The best way to build your own non-JVM lakehouse is to use Iceberg for metadata, Parquet for the Data, Query with DuckDB using Arrow tables (read Parquet directly into Arrow is very low cost), and then use Arrow->Pandas or Polars (either directly or via a service with Arrow Flight).

If you put Feather in the mix, the whole Python lakehouse stack doesn't currently work.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#14

Post by benjaminwootton »

This is a big deal in the database world as delta, iceberg and hudi mean that data is being stored in an open source format, often on S3.

It means that the storage and much of the processing is being standrdised so that you can move between databases easily and almost all tools will eventually be able to work with the same set of files in a transactionally sound way.

For instance, Snowflake could be writing to a file, a data scientist could be querying the data live from a Jupyter notebook, and ClickHouse could be serving user facing analytics against the same data with consistency guarantees.

If the business then decide to switch Snowflake to Databricks then it isn’t such a big deal.

Right now it isn’t quite as fast to query these formats on S3 as a native ingestion would be, but every database vendor will be forced by the market to optimise for performance such that they tend towards the performance of natively ingested data.

It’s a great win for openness and open source and for businesses to have their data in open and portable formats.

Lakehouse has the same implications. Lots of companies have data lakes and data warehouses and end up copying data between the two. To query the same set of data and have just one system to manage is equally impactful.

It’s a very interesting time to be in the data engineering world.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#15

Post by benjaminwootton »

Boxxed wrote: post #13

One thing I'm confused about is why does Iceberg need a spark deployment to function? Or am I wrong about that? I would rather avoid that ecosystem if I can.

It tends to be more library dependencies than live clusters.

A lot of data lakes are managed using Hadoop and Spark so I think it’s just an artefact of that.

In the end I can’t see why you wouldn’t just be able to create and manage Iceberg files directly from a standard Python/JS/Java without that legacy.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#16

Post by benjaminwootton »

debo_ wrote: post #6

I've heard of data lakes, but "data lakehouse" sounds like where upper class data goes in the summer to take their data-boats data-fishing.

The name is easy to poke fun at, but I think it’s a real problem. A lot of companies use data lakes to store data and warehouses to serve BI to tools like Tableau or PowerBI. They then up copying data between the two.

Querying a lake directly and having transactions, governance etc against one set of data (a data Lakehouse) can really simplify the stack and take out cost.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#17

Post by alexott »

jamesblonde wrote: post #10
snthpy wrote: post #8

Earlier quoted context omitted.

Thanks for this. I've been following this space for about a year or two and was wondering why Iceberg was more popular in open source. Over the past six months I got the impression that Delta is pulling ahead in the race as Iceberg is struggling to provide tools for people not in the JVM ecosystem. Delta is a lot more accessible in that way.

I guess you are referring to delta-rs (for Python in particular). An interesting factoid here is that Databricks started delta-rs, and other companies are now driving it forward - not Databricks. I guess it is not in Databricks interest to push the non JVM ecosystem. PyIceberg is catching up. Write support is almost there - https://github.com/apache/iceberg-python/pull/41

As I remember, delta-rs was started by Scribd, not by Databricks: https://youtu.be/2jgfpJD5D6U, https://youtu.be/scYz12UK-OY

Re: Understanding Parquet, Iceberg and Data Lakehouses

#19

Post by [deleted] »

benjaminwootton wrote: post #16
debo_ wrote: post #6

I've heard of data lakes, but "data lakehouse" sounds like where upper class data goes in the summer to take their data-boats data-fishing.

The name is easy to poke fun at, but I think it’s a real problem. A lot of companies use data lakes to store data and warehouses to serve BI to tools like Tableau or PowerBI. They then up copying data between the two. Querying a lake directly and having transactions, governance etc against one set of data (a data Lakehouse) can really simplify the stack and take out cost.

[deleted]

Re: Understanding Parquet, Iceberg and Data Lakehouses

#20

Post by jamesblonde »

alexott wrote: post #17
jamesblonde wrote: post #10

Earlier quoted context omitted.

I guess you are referring to delta-rs (for Python in particular). An interesting factoid here is that Databricks started delta-rs, and other companies are now driving it forward - not Databricks. I guess it is not in Databricks interest to push the non JVM ecosystem. PyIceberg is catching up. Write support is almost there - https://github.com/apache/iceberg-python/pull/41

As I remember, delta-rs was started by Scribd, not by Databricks: https://youtu.be/2jgfpJD5D6U , https://youtu.be/scYz12UK-OY

I stand corrected, then.
Post reply on HN