Understanding Parquet, Iceberg and Data Lakehouses • Hacker News Board

Re: Understanding Parquet, Iceberg and Data Lakehouses

#81

Post by oceanplexian » Sat, Dec 30, 2023, 6:28 PM UTC

Scubabear68 wrote: post #64

debo_ wrote: post #6

I've heard of data lakes, but "data lakehouse" sounds like where upper class data goes in the summer to take their data-boats data-fishing.

Naming is hard, I hope the industry can come up with something better eventually. It is definitely jarring in my head every time I hear it or read it.

I prefer the name, as I find there's a straight line correlation between stupid names like "Data Lakehouse" and bad engineering practices. Another sign is the dumber the buzzword, the more consultants and middle men exist to skim money off a problem that should never exist in the first place.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#82

Post by oceanplexian » Sat, Dec 30, 2023, 6:35 PM UTC

wokwokwok wrote: post #25

It's really easy to get lost in the technical jargon that the vendors who are selling products throw around, but this article has missed the important part, and spent all the time talking about the relatively unimportant part (data formats). You need to step back and look from a broader perspective to understand this domain. Talking about arrow/parquet/iceberg is like talking about InnoDB vs MyISAM when you're talkin…

> The problem is that generating this structured data from absolutely massive amounts of unstructured data involves a truly colossal amount of engineering work

Isn't this kind of obsolete in 2023 with LLMs?

Sure, AI is currently slow, and very expensive, but eventually the idea of needing to query a massive unstructured data source is something that will go the way of the dodo bird when you have a technology that can magically turn unstructured data into structured data quite efficiently. And in that case, when your data is properly structured, there are database technologies that are 1000x more efficient than all of these MapReduce-esque solutions for BI.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#83

Post by unknownhedgie » Sat, Dec 30, 2023, 6:37 PM UTC

lysecret wrote: post #52

I am currently working with about 100TB data on GCP with BigQuery as a query engine and simple hive partitioning like /key3=000/key2=002/. We are happy because we can run all the queries you want and it is insanely cheap. But latency is reaching quite high levels (it doesn't matter so much for us) but I was wondering, if implementing Iceberg would improve this? Has anyone experience with this? Overall this kind of ar…

We're at ~1/10th your scale but query speed is a major concern due to (rightfully) impatient clients when it comes to data viz. Unless you're using BQ tables as input for high throughput compute, I'd focus on optimizing your BI tool or creating analysis tables that would prevent end-user slowdown.

Ex: Recently created a big table (by materializing fact/dimension table joins and COALESCE operations) solely for analysis purposes. It sits "outside" our normal data warehouse setup mentally, but we can still maintain data quality/lineage as it exists inside dbt. Allowed us to do away with Tableau fixed calculations and cut load/group by times for end users ~95%.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#84

Post by pjot » Sat, Dec 30, 2023, 7:01 PM UTC

fifilura wrote: post #74

pjot wrote: post #24

Earlier quoted context omitted.

Apache Arrow and Substrait have been working towards making this a reality. I see a future where executing a query can/will send plans to many different engines distributed across the cloud, but also locally on your on machine.

It feels like you could just as well pack the runtime/engine into the job you are requesting? Am I wrong?

The point is more so in aim of creating interoperability between systems and making them in turn composable.

When there’s a common intermediate representation you can pass around those compute instructions and execute. And when there’s shared memory formats data can pass from storage to engine without serialization/deserialization.

So it wouldn’t matter if data is here or there, in this or that format, because the instructions are the same the specific interface (snowflake, MySQL, a local parquet file, etc) is irrelevant mitigating the need for glue code.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#85

Post by throwitaway222 » Sat, Dec 30, 2023, 7:31 PM UTC

Why is no one able to describe all this with more concrete ideas, like this is how you store data, this is how you connect and query - and how fast those queries will be (ie transactional speed vs "analytics" speed).

Re: Understanding Parquet, Iceberg and Data Lakehouses

#86

Post by apwell23 » Sat, Dec 30, 2023, 7:31 PM UTC

chimerasaurus wrote: post #71

apwell23 wrote: post #38

Earlier quoted context omitted.

> " every database vendor will be forced by the market to optimise for performance such that they tend towards the performance of natively ingested data." This assumes that their internal storage format has nothing to do with decades of engineering infrastructure that they built their business model around and that they would simply give all that up and compete based on just their compute layer. snowflake might as we…

I’ll just point out on the Snowflake side, we’ve been very public saying we want Iceberg/Parquet to be at or as close to parity as possible with our native format. The value add is the platform, not lock in. That also forces us to be the best on open formats, which IMO is also a good thing for everyone. Disclaimer: I work at Snowflake literally on this with my team. :)

> we’ve been very public saying we want Iceberg/Parquet to be at or as close to parity as possible with our native format

Thats great to hear. Would this mean that external iceberg tables would have the same performance as native table ? My impression of parent comment was that, eventually there would be no such thing as 'native format'. Really interested to see public statements by snowflake to that effect, would love to share that with my team.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#87

Post by zbentley » Sat, Dec 30, 2023, 7:39 PM UTC

Pasivae wrote: post #60

benjaminwootton wrote: post #14

This is a big deal in the database world as delta, iceberg and hudi mean that data is being stored in an open source format, often on S3. It means that the storage and much of the processing is being standrdised so that you can move between databases easily and almost all tools will eventually be able to work with the same set of files in a transactionally sound way. For instance, Snowflake could be writing to a file…

[flagged]

...gpt? Or ... why would you summarize an adjacent thread? Am I missing a joke?

Re: Understanding Parquet, Iceberg and Data Lakehouses

#88

Post by hermitcrab » Sat, Dec 30, 2023, 7:49 PM UTC

meehai wrote: post #29

can confirm that it is a nice thing to work with parquet files. Before this, we've worked for ~1 year with CSVs (I know the horror) and we made an effort to port all the 'legacy' code to Parquet files We interface with BigQuery (via Airflow) mostly, and except one very annoying situation it's a big improvement in terms of speed (parsing floats after querying the DB is NEVER a good option). --- In case anyone's wonder…

>Before this, we've worked for ~1 year with CSVs (I know the horror)

The CSV format (or lack of) is such a mess. You don't appreciate how much until you have to write a CSV parser and do real world support for it. Ugh.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#89

Post by hermitcrab » Sat, Dec 30, 2023, 7:54 PM UTC

pitah1 wrote: post #37

Good in-depth insights into each format. This complements nicely with a site I created called tech-diff ( https://tech-diff.com/file/ ) where it provides a summary of the file formats.

"Comma-Separated Values (CSV) is a text file format that uses commas to separate values in plain text."

Except when it uses semi-colons. Or pipes. Or something else.

Re: Understanding Parquet, Iceberg and Data Lakehouses

#90

Post by fancy_pantser » Sat, Dec 30, 2023, 8:00 PM UTC

I know it's a newcomer still under heavy development, but I'm surprised to not see Lance (and Lancedb atop it) mentioned. It crushes ORC and Parquet for most real-world data scenarios and has cheap data versioning.