What Is the Data Lakehouse Pattern?

paulddraper wrote: post #3

Hackbraten wrote: post #2

Too buzzwordy for me.

Unstructured predictive big data algorithm unsupervised deep ML internet of things mining

Is it wrong I want to invest in your pitch?

Re: What Is the Data Lakehouse Pattern?

#5

Post by benjaminwootton » Tue, Sep 14, 2021, 9:14 PM UTC

Hackbraten wrote: post #2

Too buzzwordy for me.

As I mention at the end of the article, it's definetly an almost laughable buzzword. I suspect whoever created it almost felt awkward to use it. I do however think there is concrete meaning behind it.

Databricks and Snowflake are both pulling it off, giving a combination of an RDBMs like experience and a data lake experience. If it can be pulled off, it strips away a lot of complexity in how big companies manage data.

Re: What Is the Data Lakehouse Pattern?

#6

Post by rubiquity » Tue, Sep 14, 2021, 9:15 PM UTC

blacktriangle wrote: post #4

paulddraper wrote: post #3

Earlier quoted context omitted.

Unstructured predictive big data algorithm unsupervised deep ML internet of things mining

Is it wrong I want to invest in your pitch?

Personally, I think they should invest in their pitch.

Re: What Is the Data Lakehouse Pattern?

#7

Post by gigatexal » Tue, Sep 14, 2021, 9:16 PM UTC

datalake = spark or presto on top of s3.

datelakehouse: I have no idea.

Re: What Is the Data Lakehouse Pattern?

#8

Post by mason55 » Tue, Sep 14, 2021, 9:24 PM UTC

The (I believe) original Lakehouse paper is here: http://cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf

Re: What Is the Data Lakehouse Pattern?

#9

Post by tragomaskhalos » Tue, Sep 14, 2021, 9:31 PM UTC

paulddraper wrote: post #3

Hackbraten wrote: post #2

Too buzzwordy for me.

Unstructured predictive big data algorithm unsupervised deep ML internet of things mining

Add 'blockchain' in there and my cheque will be straight in the post

Re: What Is the Data Lakehouse Pattern?

#10

Post by glogla » Tue, Sep 14, 2021, 9:33 PM UTC

"Data Lakehouse" is sadly term ruined by AWS.

It used to mean "data lake extended to support data warehouse use cases".

So something like HDFS or S3 with Delta (from DBX) or Apache Iceberg storage formats, utilizing Spark or Presto/Trino or something for compute. One unified platform built on scalable big data technologies, that can do transactions, SQL MERGE, smart partitioning and other bells and whistles.

Then AWS decided to unveil "AWS LakeHouse" which meant you have both S3 and Redshift and use both at the same time - lake and warehouse next to each other.

This is not what lakehouse meant until then It is also terrible design - having data in two places means you now have to implement access control, logging, auditing, data access and so on twice. You also have to sync data between the two storages, keep track of what is where and keep track of what is the single source of truth

Truly idiotic design / marketing that could only have come from AWS. But since any larger company has army of "enterprise architects" who went from "nobody was ever fired for recommending IBM" to "nobody was ever fired for recommending Oracle" to "nobody was ever fired for recommending AWS" who will just internally enforce whatever bullshit vendors pushes on them ... it is almost what "lakehouse" means nowadays.

AWS truly is the Oracle of 2020s. Fuck them.

(Rant over, sorry, got carried away)