41–50 of 107 posts
I've heard of data lakes, but "data lakehouse" sounds like where upper class data goes in the summer to take their data-boats data-fishing.
Sorry genuine question -- what does the phrase "at Broad" at the end of the blog post's title mean or refer to? Maybe a phrase that I am unfamiliar with? I first wondered if it is the name of an organization or team -- and this post is describing what they did in that team, but that doesn't seem to be the case? >> Understanding Parquet, Iceberg and Data Lakehouses at Broad
It's really easy to get lost in the technical jargon that the vendors who are selling products throw around, but this article has missed the important part, and spent all the time talking about the relatively unimportant part (data formats). You need to step back and look from a broader perspective to understand this domain. Talking about arrow/parquet/iceberg is like talking about InnoDB vs MyISAM when you're talkin…
All of this is correct, but it misses the main point of the new table formats - they are open-source and the data can be stored on very low cost storgae - S3. So, having a data warehouse that stores TBs or even PBs of data is not as expensive as it used to be (by an order of magnitude or more). And the formats for storing the data (Parquet), its metadata (Iceberg, Hudi, delta lake), its query engine (DuckDB, Polars,…
I didn't miss it; it's irrelevant.
It makes, almost no difference in practice, between a competent implementation in one and a competent implementation in the other.
It makes absolutely no difference that they are open source.
Understanding the details of each of the individual components will give you no meaningful insight into how to build a lakehouse.
...because, when you slap all those parts together, in whatever configuration you've picked what you end up with is a database.
A big, powerful cloud database.
Well, you have a database now and you still have zero insights and zero idea how to get any of them; that because you didn't understand that you need to build some kind of data warehouse on top of that database. You need to load the data. You need to transform the data. You need to visualize the data and build reports on it. If you're good, you probably need to preprocess the data to use as training inputs.
I'll say it more clearly and explicitly one. more. time:
- Having a database != having a data warehouse.
- Having a big cloud database build out of cloud storage, table formats, metadata engines and query engines != a lakehouse.
Having an empty database is of no value to anyone, no matter how good it is.
All of those parts, all of those things are only the first step. It's like installing postgres. Right, good job. We're done here? Reports? Oh, you can probably import something or something or I know, powerBI is good, let's install that. It'll tell you you have no data... but... we've got the infra now right? Basically done.
It's just step 1.
Parquet has been the lakehouse file format of choice for nearly half a decade. But we are starting to see other contenders that are optimized more for lower latency like lance https://github.com/lancedb/lance
I am very excited about Iceberg specifically (because open-source), but the last time I looked into it the only implementation was a Spark library, and Trino's (formerly Presto, an SQL engine) Iceberg connector had a hard dependency on Hive! It is like the entire industry had a hard time divorcing its MapReduce, Hive, and dare I to say Spark, legacy. I didn't look into Iceberg since, but plan to, and I am really look…
I often hear references to Apache Iceberg and Delta Lake as if they’re two peas in the Open Table Formats pod. Yet… Here’s the Apache Iceberg table format specification: https://iceberg.apache.org/spec/ As they like to say in patent law, anyone “skilled in the art” of database systems could use this to build and query Iceberg tables without too much difficulty. This is nominally the Delta Lake equivalent: https://git…
How do dependencies work in this type of data lakehouse? Does the orchestration layer handle that or is there metadata within the data lake that provides completeness information?
This is a big deal in the database world as delta, iceberg and hudi mean that data is being stored in an open source format, often on S3. It means that the storage and much of the processing is being standrdised so that you can move between databases easily and almost all tools will eventually be able to work with the same set of files in a transactionally sound way. For instance, Snowflake could be writing to a file…
Apache Arrow and Substrait have been working towards making this a reality. I see a future where executing a query can/will send plans to many different engines distributed across the cloud, but also locally on your on machine.
I've heard of data lakes, but "data lakehouse" sounds like where upper class data goes in the summer to take their data-boats data-fishing.
I never understood what is meant by “data lake” in the first place, other than “heterogenous collection of large-ish data files”.