Open table formats are inevitable for analytical datasets
ensembleanalytics.io
Open table formats are inevitable for analytical datasets
1–10 of 60 posts
Re: Open table formats are inevitable for analytical datasets
#2This is the best reference for comparing the 3 formats:
https://www.onehouse.ai/blog/apache-hudi-vs-delta-lake-vs-ap...
Re: Open table formats are inevitable for analytical datasets
#3Can we fix the title of this to the actual title of the blog per HN guidelines? The article isn't just about or even focused on Iceberg.
Re: Open table formats are inevitable for analytical datasets
#4Thanks for the upvotes.
Here is a more technical deep dive comparing the three main open formats - https://www.onehouse.ai/blog/apache-hudi-vs-delta-lake-vs-ap...
Re: Open table formats are inevitable for analytical datasets
#5"Open table formats" here seem to be the same as "data lakehouse" I've read other places.
It'd be great to agree on some nomenclature. Neither are very descriptive (or simple) in my view.
Re: Open table formats are inevitable for analytical datasets
#6"Open table formats" here seem to be the same as "data lakehouse" I've read other places. It'd be great to agree on some nomenclature. Neither are very descriptive (or simple) in my view.
They are closely related.
A lakehouse would be a collection of tables.
There would be some SQL engine such as Trino, Databricks, ClickHouse brokering access to these tables.
The lakehouse might be organised into layers of tables where we have raw, intermediate, processed tables.
The concept of a lakehouse doesn’t really work if you don’t have transactions and updates, which these table formats enable.
I always say, Lakehouse is a stupid name but an amazing concept and I’m sure the data industry will go this way.
Re: Open table formats are inevitable for analytical datasets
#7"Open table formats" here seem to be the same as "data lakehouse" I've read other places. It'd be great to agree on some nomenclature. Neither are very descriptive (or simple) in my view.
They are closely related. A lakehouse would be a collection of tables. There would be some SQL engine such as Trino, Databricks, ClickHouse brokering access to these tables. The lakehouse might be organised into layers of tables where we have raw, intermediate, processed tables. The concept of a lakehouse doesn’t really work if you don’t have transactions and updates, which these table formats enable. I always say, L…
Does lakehouse have the same meaning as datalake?
I ask because, if I didn't know either word, the one would mean, to me, "tiny storage next to a big body of data" and the other would mean "a big body of data".
Re: Open table formats are inevitable for analytical datasets
#8Earlier quoted context omitted.
They are closely related. A lakehouse would be a collection of tables. There would be some SQL engine such as Trino, Databricks, ClickHouse brokering access to these tables. The lakehouse might be organised into layers of tables where we have raw, intermediate, processed tables. The concept of a lakehouse doesn’t really work if you don’t have transactions and updates, which these table formats enable. I always say, L…
Does lakehouse have the same meaning as datalake? I ask because, if I didn't know either word, the one would mean, to me, "tiny storage next to a big body of data" and the other would mean "a big body of data".
"(Data) lakehouse" is an amalgamation of data warehouses and data lakes. It's meant to enable querying and all the support (transaction, etc) of traditional data warehouses on a data lake (unstructured data lying on cheap storage).
Re: Open table formats are inevitable for analytical datasets
#9Earlier quoted context omitted.
They are closely related. A lakehouse would be a collection of tables. There would be some SQL engine such as Trino, Databricks, ClickHouse brokering access to these tables. The lakehouse might be organised into layers of tables where we have raw, intermediate, processed tables. The concept of a lakehouse doesn’t really work if you don’t have transactions and updates, which these table formats enable. I always say, L…
Does lakehouse have the same meaning as datalake? I ask because, if I didn't know either word, the one would mean, to me, "tiny storage next to a big body of data" and the other would mean "a big body of data".
Date lake can be thought of a file system. Imagine 100 CSVs in folders, usually stored on S3.
Data Lakehouse involves adding things like the ability to query via SQL, the ability to update/insert/delete, transactions.
Where before people needed warehouses for BI and lakes for data science, they can now have only one approach.
It’s likely to be a big trend as data moves to this format and arrangement and the DBMS vendors like Snowflake have to play nicely with it.
Re: Open table formats are inevitable for analytical datasets
#10Thanks for the upvotes. Here is a more technical deep dive comparing the three main open formats - https://www.onehouse.ai/blog/apache-hudi-vs-delta-lake-vs-ap...
That comparison blog seems biased toward Hudi.