Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open table formats are inevitable for analytical datasets

ensembleanalytics.io

Post reply on HN
Print view View original

11–20 of 60 posts

Re: Open table formats are inevitable for analytical datasets

#11

Post by lelanthran »

benjaminwootton wrote: post #9
lelanthran wrote: post #7

Earlier quoted context omitted.

Does lakehouse have the same meaning as datalake? I ask because, if I didn't know either word, the one would mean, to me, "tiny storage next to a big body of data" and the other would mean "a big body of data".

Date lake can be thought of a file system. Imagine 100 CSVs in folders, usually stored on S3. Data Lakehouse involves adding things like the ability to query via SQL, the ability to update/insert/delete, transactions. Where before people needed warehouses for BI and lakes for data science, they can now have only one approach. It’s likely to be a big trend as data moves to this format and arrangement and the DBMS vend…

> Where before people needed warehouses for BI and lakes for data science, they can now have only one approach.

This is all very interesting, and thank you for taking the time to explain. Any good starting points for someone who would like to know more?

Re: Open table formats are inevitable for analytical datasets

#12

Post by lelanthran »

ibotty wrote: post #8
lelanthran wrote: post #7

Earlier quoted context omitted.

Does lakehouse have the same meaning as datalake? I ask because, if I didn't know either word, the one would mean, to me, "tiny storage next to a big body of data" and the other would mean "a big body of data".

"(Data) lakehouse" is an amalgamation of data warehouses and data lakes. It's meant to enable querying and all the support (transaction, etc) of traditional data warehouses on a data lake (unstructured data lying on cheap storage).

> "(Data) lakehouse" is an amalgamation of data warehouses and data lakes. It's meant to enable querying and all the support (transaction, etc) of traditional data warehouses on a data lake (unstructured data lying on cheap storage).

Thank you for that. Do you have any suggestions on where one would start if they wanted to get a better idea and/or some experience using lakehouses?

Re: Open table formats are inevitable for analytical datasets

#13

Post by benjaminwootton »

lelanthran wrote: post #11
benjaminwootton wrote: post #9

Earlier quoted context omitted.

Date lake can be thought of a file system. Imagine 100 CSVs in folders, usually stored on S3. Data Lakehouse involves adding things like the ability to query via SQL, the ability to update/insert/delete, transactions. Where before people needed warehouses for BI and lakes for data science, they can now have only one approach. It’s likely to be a big trend as data moves to this format and arrangement and the DBMS vend…

> Where before people needed warehouses for BI and lakes for data science, they can now have only one approach. This is all very interesting, and thank you for taking the time to explain. Any good starting points for someone who would like to know more?

Databricks popularised the concept and explain it very well - https://youtu.be/g11y-kJHr3I?si=j8FAkFsIjScHv24f

It’s a technology independent pattern though.

Re: Open table formats are inevitable for analytical datasets

#14

Post by 8n4vidtmkvmk »

I don't see any mention of sqlite. Is a sqlite file not the same thing they're talking about here? Pretty sure it has a spec and hasn't changed formats in many years so if you wanted to read it out in something that isn't sqlite, I imagine it wouldn't be too hard.

Re: Open table formats are inevitable for analytical datasets

#15

Post by benjaminwootton »

8n4vidtmkvmk wrote: post #14

I don't see any mention of sqlite. Is a sqlite file not the same thing they're talking about here? Pretty sure it has a spec and hasn't changed formats in many years so if you wanted to read it out in something that isn't sqlite, I imagine it wouldn't be too hard.

I agree that sqlite has a number of similar benefits - openness, table abstractions and concurrent transactions. It’s also a library so close to how delta, iceberg and hudi are implemented.

I’m glad it’s had an uptick in interest recently but I haven’t yet seen it mentioned for analytics yet.

I assume it’s row rather than column oriented?

Re: Open table formats are inevitable for analytical datasets

#16

Post by speedgoose »

benjaminwootton wrote: post #15
8n4vidtmkvmk wrote: post #14

I don't see any mention of sqlite. Is a sqlite file not the same thing they're talking about here? Pretty sure it has a spec and hasn't changed formats in many years so if you wanted to read it out in something that isn't sqlite, I imagine it wouldn't be too hard.

I agree that sqlite has a number of similar benefits - openness, table abstractions and concurrent transactions. It’s also a library so close to how delta, iceberg and hudi are implemented. I’m glad it’s had an uptick in interest recently but I haven’t yet seen it mentioned for analytics yet. I assume it’s row rather than column oriented?

Yes, SQLite is row oriented. It's not a very space efficient format because it also doesn't support compression or compact representations of numbers in binary.

But it doesn't rely on the JVM and a typical JVM ecosystem. It is a big benefit for some use-cases, like dealing with numerical data on the edge.

Re: Open table formats are inevitable for analytical datasets

#17

Post by bbuchert »

speedgoose wrote: post #16
benjaminwootton wrote: post #15

Earlier quoted context omitted.

I agree that sqlite has a number of similar benefits - openness, table abstractions and concurrent transactions. It’s also a library so close to how delta, iceberg and hudi are implemented. I’m glad it’s had an uptick in interest recently but I haven’t yet seen it mentioned for analytics yet. I assume it’s row rather than column oriented?

Yes, SQLite is row oriented. It's not a very space efficient format because it also doesn't support compression or compact representations of numbers in binary. But it doesn't rely on the JVM and a typical JVM ecosystem. It is a big benefit for some use-cases, like dealing with numerical data on the edge.

If you are searching for the analytics equivalent you should check out duckdb.

Re: Open table formats are inevitable for analytical datasets

#18

Post by hermitcrab »

benjaminwootton wrote: post #9
lelanthran wrote: post #7

Earlier quoted context omitted.

Does lakehouse have the same meaning as datalake? I ask because, if I didn't know either word, the one would mean, to me, "tiny storage next to a big body of data" and the other would mean "a big body of data".

Date lake can be thought of a file system. Imagine 100 CSVs in folders, usually stored on S3. Data Lakehouse involves adding things like the ability to query via SQL, the ability to update/insert/delete, transactions. Where before people needed warehouses for BI and lakes for data science, they can now have only one approach. It’s likely to be a big trend as data moves to this format and arrangement and the DBMS vend…

Data warehouse, data lake, data lakehouse. The data world has some terrible terminology.

Re: Open table formats are inevitable for analytical datasets

#19

Post by lalaland1125 »

8n4vidtmkvmk wrote: post #14

I don't see any mention of sqlite. Is a sqlite file not the same thing they're talking about here? Pretty sure it has a spec and hasn't changed formats in many years so if you wanted to read it out in something that isn't sqlite, I imagine it wouldn't be too hard.

The main disadvantage of sqlite compared to these other formats is that sqlite is designed for single machine processing. This can be problematic for huge datasets or complex queries.

These other formats easily support using clusters to process your data.

Re: Open table formats are inevitable for analytical datasets

#20

Post by lalaland1125 »

dikei wrote: post #10
benjaminwootton wrote: post #4

Thanks for the upvotes. Here is a more technical deep dive comparing the three main open formats - https://www.onehouse.ai/blog/apache-hudi-vs-delta-lake-vs-ap...

That comparison blog seems biased toward Hudi.

Biased in what way? The authors provide solid arguments for why they think Hudi is a good tool.
Post reply on HN