31–40 of 50 posts
As an outsider to this whole movement, data lakes have always seemed like a FOMO product. Like they've heard about big data, but they don't have much data, so they just start piling up stuff until it's "big". Also they don't know what analysis they even want to do with it, so there's no structure.
Lakehouse seems like an evolution of Hadoop to add better SQL and transactions + reasonable performance on large datasets. ("Reasonable" = not dog slow like Hive.) Reading this article as well as the survey Armbrust, Ghodsi et al. paper [0] you might easily forget that a large fraction of new data warehouse use cases get real-time data from event streams like Kafka, not S3 or HDFS. They also require stable response i…
I don't think your argument holds here at all. It's a common misconception to think high performance would require tight coupling of storage and query processing.
"Think columnar storage, high compression, vectorized query, materialized views, etc." All of those are possible in Lakehouse, and all but one (materialized views) are fully implemented on Databricks. And the remaining one isn't far away either (materialized views is really just incremental query processing + view selection, and neither problem has much to do with storage).
Lately we have moved to 'Data Pond pattern' where data store is attached directly to micro service for all CRUD operations. And of course we use Data Well pattern for deep data analysis. Another emerging architecture I see myself investigating is Data Sewer Pattern where huge load of useless data be dumped on to millions of unsuspecting entities via social media.
Handwaves swat away all technical questions. Salespeople of course turn to execs and promise magic bullet.
One month later "You're all porting your datastores to DATALAKE INTERNATIONAL SYSTEMS".
The biggest challenge I have is dealing with the fact that I now have an impressive amount of "python developers" doing whatever they can to solve their problems. I genuinely think we're improving the ability of our business to do analytics on top of our enterprise data, but sometimes I worry about the amount of technical debt we just allowed to accrue.
Data werelake - it transforms data into information every full moon. Seriously though I expect data people to understand that the value of clear descriptive naming. Communicate meaning not marketing speak.
"Data Lakehouse" is sadly term ruined by AWS. It used to mean "data lake extended to support data warehouse use cases". So something like HDFS or S3 with Delta (from DBX) or Apache Iceberg storage formats, utilizing Spark or Presto/Trino or something for compute. One unified platform built on scalable big data technologies, that can do transactions, SQL MERGE, smart partitioning and other bells and whistles. Then AWS…
> It is also terrible design - having data in two places means you now have to implement access control, logging, auditing, data access and so on twice. I've understood and implemented differently. With Spectrum (or Polybase for SQL Server / Synapse), you can extended into the data lake. Copy over aggregate/curated data or something you need to special use cases on. Leave the structured, columnar data within the chea…
With ra3 redshift, you pay storage cost of s3 for internal data as well, so unless you use the s3 with something else, I don't see much point in using spectrum.
Still, something like Snowflake works much better. They actually seem to have vision and not just "us too!" like AWS.
What does that mean?
Maybe I'm missing something, but what I need (and what the data team at my company spectacularly fails to deliver) is fast access to post-ETL data as well as pre-ETL data. I need the raw data sometimes, I need the processed data sometimes. What I get instead is no access to the raw data, and broken/slow/uninsured access to the processed data. I don't know about other people but in general I think a decent data wareho…