Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Rise and Fall of the OLAP Cube

holistics.io

Post reply on HN
Print view View original

31–40 of 54 posts

Re: The Rise and Fall of the OLAP Cube

#31

Post by beefield »

slumdev wrote: post #24
beefield wrote: post #22

Sorry, could someone ELI5 what is OLAP? And while you are there, what is Tabular Model? As background,I have worked with SQL and relational databases, and occasionally keep on hearing these, but nobody ever explained to me what these are and why I should be interested. So far I have just shrugged and thought that I guess my workloads/datamodels/whatnot just do not need these fancy things, but always I see them, there…

OLAP: Online Analytical Processing. Cranking through large amounts of data with a focus on aggregations like sums, averages, medians, etc. Measures (numbers) are defined by dimensions (attributes with usually discrete domains). Aggregations are frequently precomputed on many (or all) dimension axes so that they are immediately available. Models can be built from something as simple as a wide CSV file or as complicate…

So, when my aggregate queries/views are too slow even after indexing and tuning, then I should start to google what OLAP is? My go-to tool for this has been materialized views (or in some cases simply a new table that is refreshed every now and then). What would be the cases when OLAP is better/worse than materialized view? (based on the main article, it sounds like pretty much no other advantage for OLAP than smaller resource requirements)

Re: The Rise and Fall of the OLAP Cube

#32

Post by pmarreck »

simo7 wrote: post #13

Two false statements in this article: > ...Amazon, Airbnb, Uber and Google have rejected the data cube... Airbnb uses Druid which is essentially an OLAP cube. > BigQuery, for instance, doesn’t allow you to update data at all It's not like that anymore since several years.

> It's not like that anymore since several years.

Apparently the article's been updated to reflect that

Re: The Rise and Fall of the OLAP Cube

#33

Post by slumdev »

beefield wrote: post #31
slumdev wrote: post #24

Earlier quoted context omitted.

OLAP: Online Analytical Processing. Cranking through large amounts of data with a focus on aggregations like sums, averages, medians, etc. Measures (numbers) are defined by dimensions (attributes with usually discrete domains). Aggregations are frequently precomputed on many (or all) dimension axes so that they are immediately available. Models can be built from something as simple as a wide CSV file or as complicate…

So, when my aggregate queries/views are too slow even after indexing and tuning, then I should start to google what OLAP is? My go-to tool for this has been materialized views (or in some cases simply a new table that is refreshed every now and then). What would be the cases when OLAP is better/worse than materialized view? (based on the main article, it sounds like pretty much no other advantage for OLAP than smalle…

If your needs are well-defined in advance, you are fine with a materialized view. What I mean by "needs" is, "What questions are you trying to answer?"

OLAP's strength is that the platforms that implement it can precompute aggregations across all of your data and let you quickly answer questions that you might not have known you had.

Re: The Rise and Fall of the OLAP Cube

#34

Post by slumdev »

zurn wrote: post #27
slumdev wrote: post #24

Earlier quoted context omitted.

OLAP: Online Analytical Processing. Cranking through large amounts of data with a focus on aggregations like sums, averages, medians, etc. Measures (numbers) are defined by dimensions (attributes with usually discrete domains). Aggregations are frequently precomputed on many (or all) dimension axes so that they are immediately available. Models can be built from something as simple as a wide CSV file or as complicate…

What are typical data sizes for this? I think the OLAP term has been around a long time, some OLAP tasks of the past are probably not so huge today, I wonder if the shrunked-by-time tasks are still called OLAP or if the smaller ones are implemented differently.

It's been about 15 years since I did any OLAP work, so "big" back then was a source database measured in gigabytes with queries that took minutes to run even with a lot of optimization.

Re: The Rise and Fall of the OLAP Cube

#35

Post by tveita »

simo7 wrote: post #13

Two false statements in this article: > ...Amazon, Airbnb, Uber and Google have rejected the data cube... Airbnb uses Druid which is essentially an OLAP cube. > BigQuery, for instance, doesn’t allow you to update data at all It's not like that anymore since several years.

Druid is a column store geared towards OLAP workloads, which seems to support the article's point. It does not store its data in a "cube" layout as far as I can tell, just by time and individual columns.

Re: The Rise and Fall of the OLAP Cube

#36

Post by iblaine »

For anyone wondering, OLAP != OLAP Cubes

OLAP = A category of databases meant for analyzing data. These are eventually consistent db's, and not OLTP db's. OLAP db's include Redshift, Teradata, Snowflake, BigQuery, and others. Generally what makes a database an MPP database is partitioning compute and storage. Generally what differentiates one MPP db from another is whether or not data and compute are colocated.

OLAP Cubes = A feature built into SQL Server, that includes has its own dialect of SQL called MDX. OLAP Cubes are decreasing in popularity because you can achieve the same results through other means and less effort.

Re: The Rise and Fall of the OLAP Cube

#37

Post by lkcubing »

Thanks for sharing. Interesting write up.

While this article accurately captures the issues with traditional OLAP Cubes, it failed to recognize the latest development in this domain.

Projects like Apache Kylin, and its commercial version Kyligence, leverage modern computer architectures such as columnar storage, distributed processing, and AI optimization to build cubes over 100s of billions rows of data that covers 100s of dimensions. The performance result is unprecedented in either traditional OLAP cubes or today's MPP data warehouses. That's why the world's largest banks, retailers, insurance companies, and manufactures are turning to Kylin/Kyligence for the most challenging analytical problems.

Not to mention the rich semantic layer that modern OLAP cube technology provides, which greatly simplifies analytics architecture in the enterprises.

And, comparing columnar stores to OLAP cubes is like comparing apples to oranges. The former is a storage format and the latter is an analytical pattern. Modern OLAP cube technology like Kylin/Kyligence stores cubes in columnar stores anyway.

Re: The Rise and Fall of the OLAP Cube

#38

Post by lkcubing »

simo7 wrote: post #13

Two false statements in this article: > ...Amazon, Airbnb, Uber and Google have rejected the data cube... Airbnb uses Druid which is essentially an OLAP cube. > BigQuery, for instance, doesn’t allow you to update data at all It's not like that anymore since several years.

Amazon uses Apache Kylin, which is a modern OLAP cube technology.

Re: The Rise and Fall of the OLAP Cube

#39

Post by amylene »

alexhutcheson wrote: post #28
davedx wrote: post #26

Earlier quoted context omitted.

> Compressed column-stores hurt OLAP, because update throughput (the “on-line” in OLAP) is relatively bad. Uncompressed / array stores are quite good. But why do you want high update throughout when you're doing mostly reads? Every definition I read about OLAP says this is one of the fundamental differences, or am I misunderstanding something?

Your users want their dashboards and reports to contain data that's as up-to-date as possible. No one is happy about waiting 24 hours or more for a batch update to run if there are decisions they need to make right now. The difference vs. OLTP is that it might be acceptable for events to take minutes or even hours to be reflected in the database, vs. ~seconds or less for a transaction where the user is interacting an…

Most of the users I work with, when asked why they want more than daily batches, respond with something emotional and not business value. I had a request for real-time updates for a massive metric that is based around monthly usage.

Ya just gotta push back, and hep people understand what the trade offs are for real-time. Most people don’t need it.

Re: The Rise and Fall of the OLAP Cube

#40

Post by bsg75 »

iblaine wrote: post #36

For anyone wondering, OLAP != OLAP Cubes OLAP = A category of databases meant for analyzing data. These are eventually consistent db's, and not OLTP db's. OLAP db's include Redshift, Teradata, Snowflake, BigQuery, and others. Generally what makes a database an MPP database is partitioning compute and storage. Generally what differentiates one MPP db from another is whether or not data and compute are colocated. OLAP…

Not unique to MS SQL Server - Oracle and IBM (InfoSphere) have cube engines, and there have been independent implementations as well: https://en.wikipedia.org/wiki/Comparison_of_OLAP_servers
Post reply on HN