Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding Parquet, Iceberg and Data Lakehouses

davidgomes.com

Post reply on HN
Print view View original

61–70 of 107 posts

Re: Understanding Parquet, Iceberg and Data Lakehouses

#61

Post by bushbaba »

lysecret wrote: post #52

I am currently working with about 100TB data on GCP with BigQuery as a query engine and simple hive partitioning like /key3=000/key2=002/. We are happy because we can run all the queries you want and it is insanely cheap. But latency is reaching quite high levels (it doesn't matter so much for us) but I was wondering, if implementing Iceberg would improve this? Has anyone experience with this? Overall this kind of ar…

Iceberg won’t speed up your queries if you are using BQ native storage. It might speed up federated queries to GCS/S3

Re: Understanding Parquet, Iceberg and Data Lakehouses

#62

Post by FridgeSeal »

nerdponx wrote: post #51
wokwokwok wrote: post #25

It's really easy to get lost in the technical jargon that the vendors who are selling products throw around, but this article has missed the important part, and spent all the time talking about the relatively unimportant part (data formats). You need to step back and look from a broader perspective to understand this domain. Talking about arrow/parquet/iceberg is like talking about InnoDB vs MyISAM when you're talkin…

I think I agree with this to some extent in that it's hard for me to imagine a use case where I have a bunch of clean Parquet files, nicely partitioned, in some kind of cloud storage system. If I'm already going through the trouble of doing ELT/ETL and making a clean copy of the raw data, why would I do that in cloud storage and not in an actual database? I don't echo your dismissal of the idea because a whole lot of…

> If I'm already going through the trouble of doing ELT/ETL and making a clean copy of the raw data, why would I do that in cloud storage and not in an actual database?

Well, depends on your requirements. You can definitely go point-to-point straight into another DB.

One reason to keep data in object storage, is it gives you a sort of “db independent” storage layer. At a previous $work, we had a tiered system: data would come in from source systems (primary application db’s, marketing systems, etc), and would be serialised verbatim in structured format in S3 (layer 1). Data eng systems would then process that data-refining it, enriching it, ensuring types and schemas, etc, which would be serialised into the next tier (layer 2). At this level they’d be nice to use, so the data analysts would operate against this data in their spark notebooks.

BI and reporting, and other applications could either use data from this layer directly, or if they had special requirements, or performed computationally difficult enough tasks, we would add another layer (layer 3) for specialised workloads and presentation layers. Layer 2 and 3 data may also be synced into data warehouses like ClickHouse.

This gave us complete lineage of data (no more mystery tables, no more “where did you get this data from”, etc), and the storage itself is reasonably cheap. Many services can query these storage layers directly, so setting up views, or projections into different layouts - even for huge quantities of data- becomes feasible and achievable with no more engineering effort than a query.

Was it a lot? Yep. Would I recommend or do it everywhere? Absolutely, 100% no I would not. Was it a good fit for that org? Yeah, arguably better than they could utilise, but for them, other approaches were anaemic and fragile at best.

Could it be done simpler now? Yep, but it got the job done then haha.

Edit to add: it was also language agnostic, which was a huge win and is an understated part of these new parquet-based solutions: you’re no longer limited to “fragile python app” or “spark cluster” to interact with your data. Rust, C#/F#, various FE tools for JS/TS (cube, etc). This is a huge win because you’re not longer tied to keeping around an aging spark/hadoop cluster that has gradually encrusted more garbage into it until it’s this massive, ultra-fragile time bomb nobody dares touch that powers mass amounts of back-office-business needs.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#63

Post by orthoxerox »

layer8 wrote: post #41
debo_ wrote: post #6

I've heard of data lakes, but "data lakehouse" sounds like where upper class data goes in the summer to take their data-boats data-fishing.

I never understood what is meant by “data lake” in the first place, other than “heterogenous collection of large-ish data files”.

Yes, a typical DWH spends a lot of cycles trying to create a single consistent interpretation of the raw data, a data lake is just this raw data, plus whatever ad hoc interpretations of it your data analysts create.

A lakehouse is basically an attempt to get most of the DWH benefit by just making these ad hoc intepretations incremental.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#64

Post by Scubabear68 »

debo_ wrote: post #6

I've heard of data lakes, but "data lakehouse" sounds like where upper class data goes in the summer to take their data-boats data-fishing.

Naming is hard, I hope the industry can come up with something better eventually.

It is definitely jarring in my head every time I hear it or read it.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#65

Post by lmeyerov »

jamesblonde wrote: post #23
lmeyerov wrote: post #21

We have been excited to dig into the Iceberg era of more managed parquet storage... But they are still years behind on supporting fast GPU IO (GPUDirect/cuFile). So every time we look at bringing them to a customer for powering AI workloads... We hit that wall. It seems inevitable, more of a when vs if. Being able to have our cake & eat it too will be very cool :)

For what use case? Image data storage? For text storage, Parquet is good enough today. PyTorch Data Loader and TF Data provide multi-threaded clients that read ahead in parallel and fill up an in-memory buffer that is then transferred in/out from GPUs. I agree that S3 can be a bottleneck here. That's why we have HopsFS as a global distributed coherent NVMe cache over S3. Anyscale have been doing something similar wit…

We are trying to saturate storage->pcie->gpu cards for tasks like gpu-accelerated log analytics, and this is increasingly the bottleneck

Re: Understanding Parquet, Iceberg and Data Lakehouses

#66

Post by albert_e »

layer8 wrote: post #43
albert_e wrote: post #3

Sorry genuine question -- what does the phrase "at Broad" at the end of the blog post's title mean or refer to? Maybe a phrase that I am unfamiliar with? I first wondered if it is the name of an organization or team -- and this post is describing what they did in that team, but that doesn't seem to be the case? >> Understanding Parquet, Iceberg and Data Lakehouses at Broad

“At broad” isn’t customary English, and the author doesn’t seem to be a native speaker. It’s probably intended to mean something like “in general” or “a big-picture view”.

thanks for the clarification.

I am not a native speaker myself; but suspected the intention of the author was more or less the same -- thanks for confirming

Re: Understanding Parquet, Iceberg and Data Lakehouses

#67

Post by albert_e »

chomp5977 wrote: post #4
albert_e wrote: post #3

Sorry genuine question -- what does the phrase "at Broad" at the end of the blog post's title mean or refer to? Maybe a phrase that I am unfamiliar with? I first wondered if it is the name of an organization or team -- and this post is describing what they did in that team, but that doesn't seem to be the case? >> Understanding Parquet, Iceberg and Data Lakehouses at Broad

Not in depth, general understanding

yes that seems to be the intended meaning; though I haven't seen such usage elsewhere (I am not a native speaker of English so my exposure is limited)

thanks

Re: Understanding Parquet, Iceberg and Data Lakehouses

#68

Post by yodon »

FridgeSeal wrote: post #57
falcor84 wrote: post #49

Earlier quoted context omitted.

Real-time Bidding on query execution? The more I think about it, I believe you actually have a viable business model here.

That’s a wildly interesting idea. It open up another market too: compatible, scalable storage. Sell shovels in a gold-rush, and what better shovel than the substrate infrastructure that those bidding query engines would probably depend on.

If the queries can be executed by any provider, you are talking about a commodity product.

The business model of selling a commodity is wildly unlike the business model tech is in today.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#69

Post by nooorofe »

lysecret wrote: post #52

I am currently working with about 100TB data on GCP with BigQuery as a query engine and simple hive partitioning like /key3=000/key2=002/. We are happy because we can run all the queries you want and it is insanely cheap. But latency is reaching quite high levels (it doesn't matter so much for us) but I was wondering, if implementing Iceberg would improve this? Has anyone experience with this? Overall this kind of ar…

There is nice summary on the topic: https://aws.amazon.com/blogs/big-data/choosing-an-open-table... ("Optimizing read performance"). Those technologies primary "Data Management at Scale" but they also extend capabilities provided by raw storage formats such as parquet. So they may help you, but the question if you are really need it. I haven't worked with BigQuery, it may include [similar features](https://cloud.google.com/bigquery/docs/search-index).

You need to define what "latency" means in your case and what is "quite high levels". We are talking about analytical data storage, it is designed for efficient batch processing. To find a single record is not a primary goal of the architecture - you will need some kind of caching/indexing for fast search. Sometimes adding "limit 1" for your single record search may solve the problem.

Be sure you are using efficent data storage format as parquet, check size of the files to be sure you don't have the ["small file problem"](https://www.royalcyber.com/blog/data-services/managing-small...), then check if you are using relevant BigQuery features. And before and after those checks run "explain" on your query, if you don't use partition keys or indexed columns your search results won't be instant in any big data system.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#70

Post by nostrademons »

apwell23 wrote: post #38
benjaminwootton wrote: post #14

This is a big deal in the database world as delta, iceberg and hudi mean that data is being stored in an open source format, often on S3. It means that the storage and much of the processing is being standrdised so that you can move between databases easily and almost all tools will eventually be able to work with the same set of files in a transactionally sound way. For instance, Snowflake could be writing to a file…

> " every database vendor will be forced by the market to optimise for performance such that they tend towards the performance of natively ingested data." This assumes that their internal storage format has nothing to do with decades of engineering infrastructure that they built their business model around and that they would simply give all that up and compete based on just their compute layer. snowflake might as we…

That's the natural evolution of most tech markets. When the tech is young, proprietary companies dominate because they can control the customer experience better and deliver functionality that is simply too complex for open solutions. As the technology matures, customers start demanding interoperability, reliability, better prices, and eventually some employees "defect" from one of the big companies and start the open standards that replace their ex-employer, or an outsider reads a paper and re-implements the technology from scratch.

> Is there as good example of open standard forcing companies to give up their proprietary tech ?

UNIX -> Linux, BSD

Oracle/Sybase -> MySQL/PostgresQL

Symbolics/Lucid -> Common Lisp

Altair/Apple/Commodore/Atari -> IBM PC & clones

VMWare -> QEMU

Basically every tech that Google pioneered and then missed out on commercializing. Protobufs -> Avro/Parquet, MapReduce -> Hadoop, Flume -> Spark, Chubby -> Zookeeper, Borg -> Kubernetes, etc.

Post reply on HN