Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding Parquet, Iceberg and Data Lakehouses

davidgomes.com

Post reply on HN
Print view View original

71–80 of 107 posts

Re: Understanding Parquet, Iceberg and Data Lakehouses

#71

Post by chimerasaurus »

apwell23 wrote: post #38
benjaminwootton wrote: post #14

This is a big deal in the database world as delta, iceberg and hudi mean that data is being stored in an open source format, often on S3. It means that the storage and much of the processing is being standrdised so that you can move between databases easily and almost all tools will eventually be able to work with the same set of files in a transactionally sound way. For instance, Snowflake could be writing to a file…

> " every database vendor will be forced by the market to optimise for performance such that they tend towards the performance of natively ingested data." This assumes that their internal storage format has nothing to do with decades of engineering infrastructure that they built their business model around and that they would simply give all that up and compete based on just their compute layer. snowflake might as we…

I’ll just point out on the Snowflake side, we’ve been very public saying we want Iceberg/Parquet to be at or as close to parity as possible with our native format. The value add is the platform, not lock in. That also forces us to be the best on open formats, which IMO is also a good thing for everyone.

Disclaimer: I work at Snowflake literally on this with my team. :)

Re: Understanding Parquet, Iceberg and Data Lakehouses

#72

Post by big_whack »

nerdponx wrote: post #51
wokwokwok wrote: post #25

It's really easy to get lost in the technical jargon that the vendors who are selling products throw around, but this article has missed the important part, and spent all the time talking about the relatively unimportant part (data formats). You need to step back and look from a broader perspective to understand this domain. Talking about arrow/parquet/iceberg is like talking about InnoDB vs MyISAM when you're talkin…

I think I agree with this to some extent in that it's hard for me to imagine a use case where I have a bunch of clean Parquet files, nicely partitioned, in some kind of cloud storage system. If I'm already going through the trouble of doing ELT/ETL and making a clean copy of the raw data, why would I do that in cloud storage and not in an actual database? I don't echo your dismissal of the idea because a whole lot of…

In my experience,

* Storing large amounts like petabytes in any database is phenomenally expensive, just for the storage alone.

* For some kinds of data, like image data, databases are generally the wrong tool.

* The consumers of these kinds of systems may have really dynamic workloads. Imagine ML jobs that kick off 1K machines simultaneously to hammer your DB and read from it as fast as possible. Cloud-managed object stores have solved this scaling issue already. If you can get infrastructure you manage out of the way, you get to leverage that work. If your DB is in the middle, you're on call for it.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#73

Post by [deleted] »

yodon wrote: post #68
FridgeSeal wrote: post #57

Earlier quoted context omitted.

That’s a wildly interesting idea. It open up another market too: compatible, scalable storage. Sell shovels in a gold-rush, and what better shovel than the substrate infrastructure that those bidding query engines would probably depend on.

If the queries can be executed by any provider, you are talking about a commodity product. The business model of selling a commodity is wildly unlike the business model tech is in today.

[deleted]

Re: Understanding Parquet, Iceberg and Data Lakehouses

#74

Post by fifilura »

pjot wrote: post #24
benjaminwootton wrote: post #14

This is a big deal in the database world as delta, iceberg and hudi mean that data is being stored in an open source format, often on S3. It means that the storage and much of the processing is being standrdised so that you can move between databases easily and almost all tools will eventually be able to work with the same set of files in a transactionally sound way. For instance, Snowflake could be writing to a file…

Apache Arrow and Substrait have been working towards making this a reality. I see a future where executing a query can/will send plans to many different engines distributed across the cloud, but also locally on your on machine.

It feels like you could just as well pack the runtime/engine into the job you are requesting? Am I wrong?

Re: Understanding Parquet, Iceberg and Data Lakehouses

#75

Post by mmyrte »

plopz wrote: post #56

Are these formats appropriate for multi dimensional gridded data or are hdf/netcdf still what people use for those?

TL;DR: In climatology, I know people are using zarr. However, I think columnar storage as in parquet also merits consideration.

My thinking goes as follows: I'm trying to read chunks from n-dimensional data with a minimum of skips/random reads. For user-facing analytics and drilling down into the data, these chunks tend to be relatively few, and I'd like to have them close to one another. For high-level statistics however, I only care that the data for each chunk of work be contiguous, since I'm going to read all chunks eventually anyways.

You can reach these goals with a partitioning strategy either in HDF or zarr or parquet, but you could also reach it with blob fields in a more traditional DB, be it relational or document based or whatever. Since any storage and memory is linear, I don't care whether a row-major or column-major array is populated from a 1d vector from columnar storage with dimensionality metadata or an explicitly array based storage format; I just trust that a table with good columnar compression doesn't waste too much storage on what is implicit in (dense) array storage.

Often, I've found that even climatological data _as it pertains to a specific analytic scenario_ is actually a sparse subset of an originally dense nd-array, e.g. only looking at data over land. This has led me to advocate for more tabular approaches, but this is very domain specific.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#76

Post by 62951413 »

twoodfin wrote: post #5

I often hear references to Apache Iceberg and Delta Lake as if they’re two peas in the Open Table Formats pod. Yet… Here’s the Apache Iceberg table format specification: https://iceberg.apache.org/spec/ As they like to say in patent law, anyone “skilled in the art” of database systems could use this to build and query Iceberg tables without too much difficulty. This is nominally the Delta Lake equivalent: https://git…

If you are a Spark shop then choosing Delta over Iceberg is a no-brainer. It's simpler and perfectly integrated. Not to mention that the Spark's Delta connector can now generate Iceberg-compatible metadata too.

The choice between the two resembles the choice between Parquet and ORC circa 2016. Two formats of broadly the same power, initially biased by a particular query engine, eventually at feature parity and universally supported.

We have got a decade of experience with OSS from Databricks so doubting their "open ecosystem" status seems a little theoretical.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#77

Post by [deleted] »

apwell23 wrote: post #38
benjaminwootton wrote: post #14

This is a big deal in the database world as delta, iceberg and hudi mean that data is being stored in an open source format, often on S3. It means that the storage and much of the processing is being standrdised so that you can move between databases easily and almost all tools will eventually be able to work with the same set of files in a transactionally sound way. For instance, Snowflake could be writing to a file…

> " every database vendor will be forced by the market to optimise for performance such that they tend towards the performance of natively ingested data." This assumes that their internal storage format has nothing to do with decades of engineering infrastructure that they built their business model around and that they would simply give all that up and compete based on just their compute layer. snowflake might as we…

[deleted]

Re: Understanding Parquet, Iceberg and Data Lakehouses

#78

Post by FridgeSeal »

yodon wrote: post #68
FridgeSeal wrote: post #57

Earlier quoted context omitted.

That’s a wildly interesting idea. It open up another market too: compatible, scalable storage. Sell shovels in a gold-rush, and what better shovel than the substrate infrastructure that those bidding query engines would probably depend on.

If the queries can be executed by any provider, you are talking about a commodity product. The business model of selling a commodity is wildly unlike the business model tech is in today.

The query execution might be commodity, but the purchasers will still need to store their data somewhere, and this somewhere will need to be able to service the bandwidth and requirements of the query execution providers.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#79

Post by [deleted] »

layer8 wrote: post #41
debo_ wrote: post #6

I've heard of data lakes, but "data lakehouse" sounds like where upper class data goes in the summer to take their data-boats data-fishing.

I never understood what is meant by “data lake” in the first place, other than “heterogenous collection of large-ish data files”.

[deleted]
Post reply on HN