Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Is the Data Lakehouse Pattern?

timeflow.systems

Post reply on HN
Print view View original

11–20 of 50 posts

Re: What Is the Data Lakehouse Pattern?

#11

Post by atwebb »

glogla wrote: post #10

"Data Lakehouse" is sadly term ruined by AWS. It used to mean "data lake extended to support data warehouse use cases". So something like HDFS or S3 with Delta (from DBX) or Apache Iceberg storage formats, utilizing Spark or Presto/Trino or something for compute. One unified platform built on scalable big data technologies, that can do transactions, SQL MERGE, smart partitioning and other bells and whistles. Then AWS…

> It is also terrible design - having data in two places means you now have to implement access control, logging, auditing, data access and so on twice.

I've understood and implemented differently. With Spectrum (or Polybase for SQL Server / Synapse), you can extended into the data lake. Copy over aggregate/curated data or something you need to special use cases on. Leave the structured, columnar data within the cheap storage. You pay per scan but it is cheap (at least to a point).

Also, Databricks took the Lakehouse moniker and sprinted with it. AWS was late to the game from what I saw (at least for marketing terminology adoption).

Re: What Is the Data Lakehouse Pattern?

#14

Post by bob1029 »

Is this new lake house going to have its own pool too?

We need to go a little bit deeper. I can sense that we are just a few steps away from circling all the way back around to fancy terminology for "Postgresql installed on a big server".

Re: What Is the Data Lakehouse Pattern?

#15

Post by Zababa »

bob1029 wrote: post #14

Is this new lake house going to have its own pool too? We need to go a little bit deeper. I can sense that we are just a few steps away from circling all the way back around to fancy terminology for "Postgresql installed on a big server".

You could go pretty far with lakes and islands https://en.wikipedia.org/wiki/Recursive_islands_and_lakes#Is...

Re: What Is the Data Lakehouse Pattern?

#16

Post by bob1029 »

Zababa wrote: post #15
bob1029 wrote: post #14

Is this new lake house going to have its own pool too? We need to go a little bit deeper. I can sense that we are just a few steps away from circling all the way back around to fancy terminology for "Postgresql installed on a big server".

You could go pretty far with lakes and islands https://en.wikipedia.org/wiki/Recursive_islands_and_lakes#Is...

Fantastic. I was concerned that this might be a thing.

Re: What Is the Data Lakehouse Pattern?

#17

Post by gnabgib »

gigatexal wrote: post #7

datalake = spark or presto on top of s3. datelakehouse: I have no idea.

data lake = HDFS, GCS, Azure blob, S3, any blob store (backed by blob store, file based)

data warehouse = Oracle, SAP, BigQuery etc (backed by database, SQL interface)

data lakehouse = Spark, Presto, Databricks, Snowflake (warehouse backed by data lake)

Re: What Is the Data Lakehouse Pattern?

#18

Post by Jugurtha »

bob1029 wrote: post #14

Is this new lake house going to have its own pool too? We need to go a little bit deeper. I can sense that we are just a few steps away from circling all the way back around to fancy terminology for "Postgresql installed on a big server".

The source perpetuates this motherflowing trend: Tensorflow, Airflow, MLflow, Metaflow, KubeFlow, Timeflow.

Re: What Is the Data Lakehouse Pattern?

#19

Post by Traster »

Maybe I'm missing something, but what I need (and what the data team at my company spectacularly fails to deliver) is fast access to post-ETL data as well as pre-ETL data. I need the raw data sometimes, I need the processed data sometimes. What I get instead is no access to the raw data, and broken/slow/uninsured access to the processed data. I don't know about other people but in general I think a decent data warehouse would make me happy and the aspiration to be a datalake is what causes the data team to make me unhappy - they fail to provide the basics because they've heard too many buzzwords.

Re: What Is the Data Lakehouse Pattern?

#20

Post by gnabgib »

mason55 wrote: post #8

The (I believe) original Lakehouse paper is here: http://cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf

Seems unlikely, Databricks used the term in a blog post January/2020[0], while the linked paper suggests it was published January/2021. Edit: Given databricks was involved in this paper, perhaps this is the first.. although both Databricks and Snowflake (and to a lesser extent Google which favours lake/warehouse convergence) have been using the term for a number of years

[0]: https://databricks.com/blog/2020/01/30/what-is-a-data-lakeho...

[1]: https://cloud.google.com/blog/products/data-analytics/data-l...

Post reply on HN