Understanding Parquet, Iceberg and Data Lakehouses • Hacker News Board

Re: Understanding Parquet, Iceberg and Data Lakehouses

#51

Post by nerdponx » Sat, Dec 30, 2023, 1:55 PM UTC

wokwokwok wrote: post #25

It's really easy to get lost in the technical jargon that the vendors who are selling products throw around, but this article has missed the important part, and spent all the time talking about the relatively unimportant part (data formats). You need to step back and look from a broader perspective to understand this domain. Talking about arrow/parquet/iceberg is like talking about InnoDB vs MyISAM when you're talkin…

I think I agree with this to some extent in that it's hard for me to imagine a use case where I have a bunch of clean Parquet files, nicely partitioned, in some kind of cloud storage system.

If I'm already going through the trouble of doing ELT/ETL and making a clean copy of the raw data, why would I do that in cloud storage and not in an actual database?

I don't echo your dismissal of the idea because a whole lot of people seem to be excited about it. But I personally feel like I'm missing the use case compared to the lake + warehouse setup.

Is it about distributing responsibility across teams? Reducing storage cost? Open source good vibes?

Maybe a legitimate use case is being able to use the same data source for multiple query engine frontends? That is, you can use both Spark and Snowflake on the same physical data files.

I'd be interested to hear about this from someone who's using or planning to use a lakehouse.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#52

Post by lysecret » Sat, Dec 30, 2023, 2:08 PM UTC

I am currently working with about 100TB data on GCP with BigQuery as a query engine and simple hive partitioning like /key3=000/key2=002/. We are happy because we can run all the queries you want and it is insanely cheap. But latency is reaching quite high levels (it doesn't matter so much for us) but I was wondering, if implementing Iceberg would improve this? Has anyone experience with this?

Overall this kind of architecture is just awesome.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#53

Post by towelpluswater » Sat, Dec 30, 2023, 2:26 PM UTC

wokwokwok wrote: post #25

It's really easy to get lost in the technical jargon that the vendors who are selling products throw around, but this article has missed the important part, and spent all the time talking about the relatively unimportant part (data formats). You need to step back and look from a broader perspective to understand this domain. Talking about arrow/parquet/iceberg is like talking about InnoDB vs MyISAM when you're talkin…

I think the bigger problem is that replication/ingestion (ie: what fivetran does) has come to represent 'ELT'. Likely by design.

And you don't need that pesky transformation part.

Except you really do, when you get beyond having a source system or two.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#54

Post by Lyngbakr » Sat, Dec 30, 2023, 2:27 PM UTC

> However, this blog post won’t be 100% comprehensive, or even the best starting point for most people. That’s because I’m writing this for myself. I find that the best way to learn new things is by "forcing myself" to re-explain them to others.

I really like this attitude and have started embracing it myself both on paper and in notes on my website.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#55

Post by FridgeSeal » Sat, Dec 30, 2023, 2:47 PM UTC

apwell23 wrote: post #38

benjaminwootton wrote: post #14

This is a big deal in the database world as delta, iceberg and hudi mean that data is being stored in an open source format, often on S3. It means that the storage and much of the processing is being standrdised so that you can move between databases easily and almost all tools will eventually be able to work with the same set of files in a transactionally sound way. For instance, Snowflake could be writing to a file…

> " every database vendor will be forced by the market to optimise for performance such that they tend towards the performance of natively ingested data." This assumes that their internal storage format has nothing to do with decades of engineering infrastructure that they built their business model around and that they would simply give all that up and compete based on just their compute layer. snowflake might as we…

> snowflake might as well shutup shop and return billions to the investors.

I mean, we can dream right?

There’s a bunch of companies that I don’t believe deserve their status or valuation and Snowflake is one of them.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#56

Post by plopz » Sat, Dec 30, 2023, 2:48 PM UTC

Are these formats appropriate for multi dimensional gridded data or are hdf/netcdf still what people use for those?

Re: Understanding Parquet, Iceberg and Data Lakehouses

#57

Post by FridgeSeal » Sat, Dec 30, 2023, 2:49 PM UTC

falcor84 wrote: post #49

pjot wrote: post #24

Earlier quoted context omitted.

Apache Arrow and Substrait have been working towards making this a reality. I see a future where executing a query can/will send plans to many different engines distributed across the cloud, but also locally on your on machine.

Real-time Bidding on query execution? The more I think about it, I believe you actually have a viable business model here.

That’s a wildly interesting idea.

It open up another market too: compatible, scalable storage. Sell shovels in a gold-rush, and what better shovel than the substrate infrastructure that those bidding query engines would probably depend on.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#58

Post by FridgeSeal » Sat, Dec 30, 2023, 2:53 PM UTC

hacful-tonteg wrote: post #32

twoodfin wrote: post #5

I often hear references to Apache Iceberg and Delta Lake as if they’re two peas in the Open Table Formats pod. Yet… Here’s the Apache Iceberg table format specification: https://iceberg.apache.org/spec/ As they like to say in patent law, anyone “skilled in the art” of database systems could use this to build and query Iceberg tables without too much difficulty. This is nominally the Delta Lake equivalent: https://git…

Microsoft is using Delta for their Fabric Lakehouse architecture and its also what OneLake is built around so now you have another massive player choosing Delta.

That’s…not exactly a winning point for Delta lake IMO.

Massive corp, with their own opaque interests and endless bodies to throw at problems has picked a favourite. That favourite being an “open” format controlled by another opaque enterprise company. I’d half expect M$ to just take it wholesale, and start modifying it to suit their own ends, until eventually the “open source” component is some skin-deep façade that completely and utterly dependent on M$ infra.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#59

Post by aejm » Sat, Dec 30, 2023, 2:57 PM UTC

I really liked your article.

Is this a typo: “Hive, Delta Lake and Iceberg all support support of schema registry or metastore.”?

Re: Understanding Parquet, Iceberg and Data Lakehouses

#60

Post by Pasivae » Sat, Dec 30, 2023, 2:58 PM UTC

benjaminwootton wrote: post #14

This is a big deal in the database world as delta, iceberg and hudi mean that data is being stored in an open source format, often on S3. It means that the storage and much of the processing is being standrdised so that you can move between databases easily and almost all tools will eventually be able to work with the same set of files in a transactionally sound way. For instance, Snowflake could be writing to a file…

[flagged]