Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Is the Data Lakehouse Pattern?

timeflow.systems

Post reply on HN
Print view View original

21–30 of 50 posts

Re: What Is the Data Lakehouse Pattern?

#21

Post by Spivak »

bob1029 wrote: post #14

Is this new lake house going to have its own pool too? We need to go a little bit deeper. I can sense that we are just a few steps away from circling all the way back around to fancy terminology for "Postgresql installed on a big server".

Isn’t the point of building stuff on top of blob stores is that it’s too much data to be housed in normal RDBMSs and the performance expectations are vastly different (run this report that needs to return results in days/weeks not ms) so you can go way way cheaper and slower with the storage?

Re: What Is the Data Lakehouse Pattern?

#22

Post by marcos100 »

gnabgib wrote: post #20
mason55 wrote: post #8

The (I believe) original Lakehouse paper is here: http://cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf

Seems unlikely, Databricks used the term in a blog post January/2020[0], while the linked paper suggests it was published January/2021. Edit: Given databricks was involved in this paper, perhaps this is the first.. although both Databricks and Snowflake (and to a lesser extent Google which favours lake/warehouse convergence) have been using the term for a number of years [0]: https://databricks.com/blog/2020/01/30/wh…

Same authors.

Re: What Is the Data Lakehouse Pattern?

#23

Post by fredliu »

It's been increasingly hard to differentiate legitimate tech architecture/patterns that helps under certain scenarios from marketing shenanigans packaged to sell you more of the same stuff. IMO, it is just natural evolution for such "lake house" patterns to emerge to address shortcomings of "data lakes" (another term passed its prime in terms of hypes around it). But the overhyped term "lakehouse" itself is actually hurting adoption, because it makes it sound like just marketing talks.

Re: What Is the Data Lakehouse Pattern?

#24

Post by coredog64 »

Jugurtha wrote: post #18
bob1029 wrote: post #14

Is this new lake house going to have its own pool too? We need to go a little bit deeper. I can sense that we are just a few steps away from circling all the way back around to fancy terminology for "Postgresql installed on a big server".

The source perpetuates this motherflowing trend: Tensorflow, Airflow, MLflow, Metaflow, KubeFlow, Timeflow.

Costflow

Re: What Is the Data Lakehouse Pattern?

#26

Post by fredliu »

bob1029 wrote: post #14

Is this new lake house going to have its own pool too? We need to go a little bit deeper. I can sense that we are just a few steps away from circling all the way back around to fancy terminology for "Postgresql installed on a big server".

Yes, there'll be an option to upgrade your "Data Swimming Pool" into a "Data Infinity Pool" overseeing your own "Data Lake", with this extra enterprise feature for only $30k/month! /s

--edit: no idea why my post struck a wrong chord somewhere. Looks like the parent comment was not meant as a joke?

Re: What Is the Data Lakehouse Pattern?

#27

Post by hodgesrm »

Lakehouse seems like an evolution of Hadoop to add better SQL and transactions + reasonable performance on large datasets. ("Reasonable" = not dog slow like Hive.) Reading this article as well as the survey Armbrust, Ghodsi et al. paper [0] you might easily forget that a large fraction of new data warehouse use cases get real-time data from event streams like Kafka, not S3 or HDFS. They also require stable response in small numbers of milliseconds for the more demanding use cases.

So Lakehouse is not really an evolution of data warehouses or at least new ones like ClickHouse and Druid. SQL data warehouses are highly optimized for analytic query speed. Think columnar storage, high compression, vectorized query, materialized views, etc. They also couple well with event streams. You can't get high performance without optimized storage and very tight integration of parts.

I have massive respect for Ali and Matei but there's no way Lakehouse will replace this.

[0] http://cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf

Edit: replaced "original" with "survey".

Re: What Is the Data Lakehouse Pattern?

#28

Post by mr_toad »

Spivak wrote: post #21
bob1029 wrote: post #14

Is this new lake house going to have its own pool too? We need to go a little bit deeper. I can sense that we are just a few steps away from circling all the way back around to fancy terminology for "Postgresql installed on a big server".

Isn’t the point of building stuff on top of blob stores is that it’s too much data to be housed in normal RDBMSs and the performance expectations are vastly different (run this report that needs to return results in days/weeks not ms) so you can go way way cheaper and slower with the storage?

Partly. It’s also because data modelling and cleaning can cause data warehouse implementations to drag on for years.

Re: What Is the Data Lakehouse Pattern?

#30

Post by geodel »

Lately we have moved to 'Data Pond pattern' where data store is attached directly to micro service for all CRUD operations. And of course we use Data Well pattern for deep data analysis.

Another emerging architecture I see myself investigating is Data Sewer Pattern where huge load of useless data be dumped on to millions of unsuspecting entities via social media.

Post reply on HN