Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open table formats are inevitable for analytical datasets

ensembleanalytics.io

Post reply on HN
Print view View original

41–50 of 60 posts

Re: Open table formats are inevitable for analytical datasets

#41

Post by datadrivenangel »

The one and a half downsides not mentioned here:

1: Separating storage and compute tends to result in relatively high startup latency (Databricks, here's looking at you!) when you have to provision new compute. For a massive batch job, waiting 10 minutes for a cluster to come up is fine, but a lot of orgs don't realize the cost implications and end up with tiny developer/analyst clusters, or aggressively spin inactive clusters down, which results in long wait times. Analyst/developer ergonomics have not traditionally been a major concern in the big data space.

1.5: Dealing with multiple files instead of a data warehouse SQL interface requires work, and can introduce interesting performance issues. Obviously you can put a SQL interface in front of the data, but most RDBMS / Data Warehouses have a lot of functionality around maintaining your data that you may not get with the native file format, so you get soft-locked into the metadata format that comes from your data lake file format. It's all open, so this is only a half issue, but there are switching costs.

Re: Open table formats are inevitable for analytical datasets

#42

Post by atwebb »

lelanthran wrote: post #7
benjaminwootton wrote: post #6

Earlier quoted context omitted.

They are closely related. A lakehouse would be a collection of tables. There would be some SQL engine such as Trino, Databricks, ClickHouse brokering access to these tables. The lakehouse might be organised into layers of tables where we have raw, intermediate, processed tables. The concept of a lakehouse doesn’t really work if you don’t have transactions and updates, which these table formats enable. I always say, L…

Does lakehouse have the same meaning as datalake? I ask because, if I didn't know either word, the one would mean, to me, "tiny storage next to a big body of data" and the other would mean "a big body of data".

That's what it originally meant, at least in my experience. It was when warehouses got access to commodity storage through virtualization options (Hey! I can read S3 from Redshift and it looks like a Redshift table). Similar to Postgres foreign data wrappers or polybase in sql server.

Databricks (with Delta as the underpinning) seems to have lead the charge of lakehouse meaning, your data lake+file formats/helpers+compute==data lake+datawarehouse==lakehouse.

The latter seems to be the prevailing definition today with the former aging in place.

Re: Open table formats are inevitable for analytical datasets

#43

Post by atwebb »

lalaland1125 wrote: post #20
dikei wrote: post #10

Earlier quoted context omitted.

That comparison blog seems biased toward Hudi.

Biased in what way? The authors provide solid arguments for why they think Hudi is a good tool.

I think it is also wrong in the capabilities, example: Redshift should be able to read Iceberg via Redshift Spectrum.

https://docs.aws.amazon.com/redshift/latest/dg/querying-iceb...

Re: Open table formats are inevitable for analytical datasets

#44

Post by CyberDildonics »

hermitcrab wrote: post #18
benjaminwootton wrote: post #9

Earlier quoted context omitted.

Date lake can be thought of a file system. Imagine 100 CSVs in folders, usually stored on S3. Data Lakehouse involves adding things like the ability to query via SQL, the ability to update/insert/delete, transactions. Where before people needed warehouses for BI and lakes for data science, they can now have only one approach. It’s likely to be a big trend as data moves to this format and arrangement and the DBMS vend…

Data warehouse, data lake, data lakehouse. The data world has some terrible terminology.

You can't resell someone a file system if you don't rename it to a "data lake". You can't resell someone an indexed file system from the 80's that can be queried with SQL unless you rename it a "data lakehouse".

Re: Open table formats are inevitable for analytical datasets

#45

Post by chimerasaurus »

benjaminwootton wrote: post #4

Thanks for the upvotes. Here is a more technical deep dive comparing the three main open formats - https://www.onehouse.ai/blog/apache-hudi-vs-delta-lake-vs-ap...

I talk to customers basically all day about table formats. Only one customer has really brought up Hudi in a meaningful way. IMO, Hudi is basically out of contention for 95%+ of people looking at table formats.

Re: Open table formats are inevitable for analytical datasets

#46

Post by chimerasaurus »

codeaken wrote: post #30
the_duke wrote: post #27

Is there any such open source table format for row based data? I've been dabbling with implementing a database, and adopting a format would be much easier.

Apache AVRO [1] is one but it has been largely replaced by Apache Parquet [2] which is a hybrid row/columnar format [1] https://avro.apache.org/ [2] https://parquet.apache.org/

As a note, Iceberg also supports AVRO in addition to Parquet (and ORC).

Re: Open table formats are inevitable for analytical datasets

#47

Post by chimerasaurus »

benjaminwootton wrote: post #9
lelanthran wrote: post #7

Earlier quoted context omitted.

Does lakehouse have the same meaning as datalake? I ask because, if I didn't know either word, the one would mean, to me, "tiny storage next to a big body of data" and the other would mean "a big body of data".

Date lake can be thought of a file system. Imagine 100 CSVs in folders, usually stored on S3. Data Lakehouse involves adding things like the ability to query via SQL, the ability to update/insert/delete, transactions. Where before people needed warehouses for BI and lakes for data science, they can now have only one approach. It’s likely to be a big trend as data moves to this format and arrangement and the DBMS vend…

Disclaimer - work at Snowflake. Two quick points to mention.

1. Snowflake has always used blob stores + file data + metadata. Architecturally it’s actually always been very Lakehouse-y

2. Parquet and Iceberg should be equivalent in performance and features. It’s more than playing nicely - it’s more choose your own adventure where all things are equal.

Re: Open table formats are inevitable for analytical datasets

#48

Post by chimerasaurus »

alpdhuez77 wrote: post #2

This is the best reference for comparing the 3 formats: https://www.onehouse.ai/blog/apache-hudi-vs-delta-lake-vs-ap...

It’s written by a group -really- trying to make one of them a thing, even though it’s in decline, so just have that lens for anyone reading it.

Re: Open table formats are inevitable for analytical datasets

#49

Post by prpl »

8n4vidtmkvmk wrote: post #14

I don't see any mention of sqlite. Is a sqlite file not the same thing they're talking about here? Pretty sure it has a spec and hasn't changed formats in many years so if you wanted to read it out in something that isn't sqlite, I imagine it wouldn't be too hard.

These are metadata layers on top of file formats, expecting to process many files (where schema may change) in typically object storage, with volumes in excess of a petabyte. There is nothing preventing you from using sqlite with iceberg, for example - except implementing it (may involve implementation in your execution engine too). It already supports Parquet, Orc, and Avro (which is more row oriented than column oriented)

Iceberg and Sqlite might be interesting if you wanted to colocate two tables in the same file, for example. A smart enough data access layer with an appropriate execution engine could possibly see the data for both.

Re: Open table formats are inevitable for analytical datasets

#50

Post by sys13 »

datadrivenangel wrote: post #41

The one and a half downsides not mentioned here: 1: Separating storage and compute tends to result in relatively high startup latency (Databricks, here's looking at you!) when you have to provision new compute. For a massive batch job, waiting 10 minutes for a cluster to come up is fine, but a lot of orgs don't realize the cost implications and end up with tiny developer/analyst clusters, or aggressively spin inactiv…

(I do some training for Databricks) 1. Yeah, cluster startup time can be not fun. Here are some solutions: - pools (keeps instances around so you don't have to wait for the cloud to provision them - serverless SQL warehouses (viable if you're doing only SQL) - one job with multiple tasks that share the same job cluster. Delta Live Tables does a similar thing but with streaming autoscaling - streaming: cluster never needs to go down. Can share multiple streams on the same cluster so they load balance each other
Post reply on HN