Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Is the Data Lakehouse Pattern?

timeflow.systems

Post reply on HN
Print view View original

41–50 of 50 posts

Re: What Is the Data Lakehouse Pattern?

#41

Post by twic »

geodel wrote: post #30

Lately we have moved to 'Data Pond pattern' where data store is attached directly to micro service for all CRUD operations. And of course we use Data Well pattern for deep data analysis. Another emerging architecture I see myself investigating is Data Sewer Pattern where huge load of useless data be dumped on to millions of unsuspecting entities via social media.

I'm sure you know that a crucial pattern during corporate restructuring is the Data Three Gorges Dam, where in the course of construction, hundreds of perfectly productive products staffed by thousands of developers will be wiped away.

Re: What Is the Data Lakehouse Pattern?

#42

Post by hodgesrm »

rxin wrote: post #32
hodgesrm wrote: post #27

Lakehouse seems like an evolution of Hadoop to add better SQL and transactions + reasonable performance on large datasets. ("Reasonable" = not dog slow like Hive.) Reading this article as well as the survey Armbrust, Ghodsi et al. paper [0] you might easily forget that a large fraction of new data warehouse use cases get real-time data from event streams like Kafka, not S3 or HDFS. They also require stable response i…

Co-author of the paper here. I don't think your argument holds here at all. It's a common misconception to think high performance would require tight coupling of storage and query processing. "Think columnar storage, high compression, vectorized query, materialized views, etc." All of those are possible in Lakehouse, and all but one (materialized views) are fully implemented on Databricks. And the remaining one isn't…

Thanks for your comment and sorry if I was unclear. I'm not arguing that storage and compute need to be directly coupled. However, storage does need to be very carefully optimized to match compute, especially when you are trying read events and make them available for immediate storage. ClickHouse for example has multiple formats for table parts in order to allow efficient buffering of rapidly arriving records. Using customized formats has allowed the project to evolve quickly.

In fact the Lakehouse paper seems to be setting up a strawman. Here are three examples.

* The new low-latency SQL data warehouses are open source. They are are not locking data in proprietary formats. We're not Snowflake.

* SQL data warehouses are already headed toward support for object storage for the same reason everyone else is: costs and durability in large datasets. Here's just one sample of many: https://altinity.com/blog/tips-for-high-performance-clickhou...

* Not everyone cares about ML and data warehouse integration. From my experience working on ClickHouse only a small percentage of users integrate ML. By contrast 100% of our users care about efficient visualization and keeping data pipelines as short as possible, hence the benefit of a tightly integrated server.

I think there's actually a bifurcation of the market into low-latency use cases driven by event streams versus much larger datasets containing unstructured/semi-structured data stored in low-cost object storage. Lakehouse addresses the latter. SQL data warehouses are focused on the former. I don't see one "winning"--both markets are growing.

Re: What Is the Data Lakehouse Pattern?

#43

Post by CRConrad »

rpedela wrote: post #36

I think using a data warehouse as your data lake or lake house is optimal. Even for data that isn't relational. Storage is so cheap now and is decoupled from compute costs for several providers that I don't even give it a thought. You get a fast, scalable SQL interface which is still nice and useful for non-relational data. Then all, or most, of the transformations needed for analysis can be pure SQL using a tool lik…

> pure SQL using a tool like DBT

I don't get it... Looks to me like DBT is a Python SQL wrapper / big library that among other things includes an SQL generator / something else like that -- but not "pure" SQL?

Re: What Is the Data Lakehouse Pattern?

#44

Post by atwebb »

glogla wrote: post #38
atwebb wrote: post #11

Earlier quoted context omitted.

> It is also terrible design - having data in two places means you now have to implement access control, logging, auditing, data access and so on twice. I've understood and implemented differently. With Spectrum (or Polybase for SQL Server / Synapse), you can extended into the data lake. Copy over aggregate/curated data or something you need to special use cases on. Leave the structured, columnar data within the chea…

You can do "lakehouse" just with Redshift, but in AWS pictures, you'll see Glue Jobs, Glue Elastic Views, Sagemaker, Aurora ... it's a huge mess. With ra3 redshift, you pay storage cost of s3 for internal data as well, so unless you use the s3 with something else, I don't see much point in using spectrum. Still, something like Snowflake works much better. They actually seem to have vision and not just "us too!" like…

You pay the storage cost in S3 as well, depending on tier Snowflake will not necessarily be a cost savings from compute either. Redshift could really use some elasticity beyond a factor of 2 and some warm resume features.

Re: What Is the Data Lakehouse Pattern?

#45

Post by hodgesrm »

rxin wrote: post #32
hodgesrm wrote: post #27

Lakehouse seems like an evolution of Hadoop to add better SQL and transactions + reasonable performance on large datasets. ("Reasonable" = not dog slow like Hive.) Reading this article as well as the survey Armbrust, Ghodsi et al. paper [0] you might easily forget that a large fraction of new data warehouse use cases get real-time data from event streams like Kafka, not S3 or HDFS. They also require stable response i…

Co-author of the paper here. I don't think your argument holds here at all. It's a common misconception to think high performance would require tight coupling of storage and query processing. "Think columnar storage, high compression, vectorized query, materialized views, etc." All of those are possible in Lakehouse, and all but one (materialized views) are fully implemented on Databricks. And the remaining one isn't…

p.s., If anyone wants to argue this point we're doing a conference on open source analytics on November 2. It's called OSA Con and the CFP is here: https://altinity.com/osa-con-2021/. It's non-partisan and free. We love all open source projects. :)

I was already thinking it would be great to get a lakehouse presentation. If you are interested please submit a proposal!!

Re: What Is the Data Lakehouse Pattern?

#46

Post by LaserToy »

I manage data infrastructure. Recently, I found I don’t know how to name What we do as many the names are marketing and embarrassing BuzzWords.

All we need is a database that can do analytical queries (and ideally OLTP) and can scale. We don’t need lakes, ponds, swamps, lake houses, sparks, …

Big Query got it right.

Re: What Is the Data Lakehouse Pattern?

#47

Post by rpedela »

CRConrad wrote: post #43
rpedela wrote: post #36

I think using a data warehouse as your data lake or lake house is optimal. Even for data that isn't relational. Storage is so cheap now and is decoupled from compute costs for several providers that I don't even give it a thought. You get a fast, scalable SQL interface which is still nice and useful for non-relational data. Then all, or most, of the transformations needed for analysis can be pure SQL using a tool lik…

> pure SQL using a tool like DBT I don't get it... Looks to me like DBT is a Python SQL wrapper / big library that among other things includes an SQL generator / something else like that -- but not "pure" SQL?

DBT has two main innovations. First, everything is a SELECT statement and DBT handles all the DDL for you. You can handle DDL yourself if you have a special case too. Second, the ref/source macros build a DAG of all your models so you don't have to think about build order. There are other innovations but those are the main ones.

You can give it truly pure SQL in both models and scripts, and mixing in Jinja if you need it for dynamic models. But I'd recommend at least using ref/source.

Re: What Is the Data Lakehouse Pattern?

#48

Post by glogla »

atwebb wrote: post #44
glogla wrote: post #38

Earlier quoted context omitted.

You can do "lakehouse" just with Redshift, but in AWS pictures, you'll see Glue Jobs, Glue Elastic Views, Sagemaker, Aurora ... it's a huge mess. With ra3 redshift, you pay storage cost of s3 for internal data as well, so unless you use the s3 with something else, I don't see much point in using spectrum. Still, something like Snowflake works much better. They actually seem to have vision and not just "us too!" like…

You pay the storage cost in S3 as well, depending on tier Snowflake will not necessarily be a cost savings from compute either. Redshift could really use some elasticity beyond a factor of 2 and some warm resume features.

Just to clarify, I meant Redshift ra3 storage costs as much as s3, so you're not saving much by keeping things in s3 instead of in Redshift.

Although I don't know how Redshift compression compares to something like gzipped parquet. Maybe the data ends up taking more space and thus money.

Agreed on that elasticity.

Re: What Is the Data Lakehouse Pattern?

#49

Post by CRConrad »

rpedela wrote: post #47
CRConrad wrote: post #43

Earlier quoted context omitted.

> pure SQL using a tool like DBT I don't get it... Looks to me like DBT is a Python SQL wrapper / big library that among other things includes an SQL generator / something else like that -- but not "pure" SQL?

DBT has two main innovations. First, everything is a SELECT statement and DBT handles all the DDL for you. You can handle DDL yourself if you have a special case too. Second, the ref/source macros build a DAG of all your models so you don't have to think about build order. There are other innovations but those are the main ones. You can give it truly pure SQL in both models and scripts, and mixing in Jinja if you nee…

Still not sure I got all that -- gotta look into it some more -- but maybe I know a little more now. Thanks!
Post reply on HN