Live data from Hacker News

Show HN: Open-Source Business Intelligence for BigQuery – Looker Alternative

mprove.io

41–46 of 46 posts

Re: Show HN: Open-Source Business Intelligence for BigQuery – Looker Alternative

#41
post #40

Earlier quoted context omitted.

LookML supports different databases as sources. Sometimes Looker adds its own LookML parameters for database - which makes it difficult to read the documentation. For analysts who work with SQL queries, any wrapper on a language seems complicated. But in the case of LookML and BlockML, in a couple of days you already begin to understand how this works, and after a week you can not tear yourself away from using the pr…

@akalitenya the way I used to explain it to the technical audience is: would you rather program in a low-level language like C or in a high-level language like Java, Python, or Node? Well, if you are writing compilers, you might choose C, but if you are doing almost everything else, you want to choose a language that fits the purpose: python for data science, Node for web backend, etc. Same with LookML - it provides…

@segah Good point. Based on your consulting experience, can you tell if there have been any cases when your clients chose another Business Intelligence tool instead of Looker, what are these tools and what are the reasons?

Re: Show HN: Open-Source Business Intelligence for BigQuery – Looker Alternative

#42
post #28

Earlier quoted context omitted.

wow, impressive! how about symmetric aggregates (e.g. being able to do correct summation/aggregation on numeric values despite a one_to_many join)?

Yes, look at this page - https://mprove.io/docs/blockml/fields/measure . You need to specify measure "type" and "sql_key" that will be used to avoid counting duplicates.

How do the underlying queries look for symmetric aggregates? Sadly SQL never supported the ability to compute an aggregate based on the unique values of another column.

Re: Show HN: Open-Source Business Intelligence for BigQuery – Looker Alternative

#43

Earlier quoted context omitted.

Yes, look at this page - https://mprove.io/docs/blockml/fields/measure . You need to specify measure "type" and "sql_key" that will be used to avoid counting duplicates.

How do the underlying queries look for symmetric aggregates? Sadly SQL never supported the ability to compute an aggregate based on the unique values of another column.

Mprove does it the way the Looker did it before:

   CREATE TEMPORARY FUNCTION mprove_array_sum(ar ARRAY) AS
  ((SELECT SUM(CAST(REGEXP_EXTRACT(val, '\\|\\|(\\-?\\d+(?:.\\d+)?)$') AS FLOAT64)) FROM UNNEST(ar) as val));
...

   SELECT COALESCE(mprove_array_sum(ARRAY_AGG(DISTINCT CONCAT(CONCAT(CAST(a.id AS STRING), '||'), CAST(a.population AS STRING)))), 0) as a_cohort_size
Recently, Looker began to do it differently, most likely to improve bigquery performance:

  COALESCE(ROUND(COALESCE(CAST( ( SUM(DISTINCT (CAST(ROUND(COALESCE(lesson_5_cohorts.population ,0)*(1/1000*1.0), 9) AS NUMERIC) + (cast(cast(concat('0x', substr(to_hex(md5(CAST(lesson_5_cohorts.id  AS STRING))), 1, 15)) as int64) as numeric) * 4294967296 + cast(cast(concat('0x', substr(to_hex(md5(CAST(lesson_5_cohorts.id  AS STRING))), 16, 8)) as int64) as numeric)) * 0.000000001 )) - SUM(DISTINCT (cast(cast(concat('0x', substr(to_hex(md5(CAST(lesson_5_cohorts.id  AS STRING))), 1, 15)) as int64) as numeric) * 4294967296 + cast(cast(concat('0x', substr(to_hex(md5(CAST(lesson_5_cohorts.id  AS STRING))), 16, 8)) as int64) as numeric)) * 0.000000001) )  / (1/1000*1.0) AS FLOAT64), 0), 6), 0) AS lesson_5_cohorts_m_sum_distinct

Re: Show HN: Open-Source Business Intelligence for BigQuery – Looker Alternative

#44

Earlier quoted context omitted.

The MySQL image specified in the docker-compose file you mentioned. It is used for the internal data of the Mprove application (users, projects, members, etc.). Each user action in the web client (angular) can initiate several queries to this database through an backend request. Delays here are crucial. network latency - so you need to keep the database as close as possible to your server side, read / write delays -…

Hmm..I would prefer to not have it. In production, managing database persistence is very hard. Especially when you go down the kubernetes road. I would take higher latency, but avoid pulling in a whole database infrastructure. Plus a huge number of us use postgresql..so that becomes another set of a mess. I would strongly urge you to do this on the same bigquery database that you would connect to anyways.

Bigquery exists only in Google Cloud. This is a columnar database like Redshift. It is not designed for fast processing small queries that are necessary to support the operation of any application. Its main feature is that it can scale the execution of heavy analytical queries between 10,000 nodes transparently for the end user. Users have no control over Bigquery instances. This is super cost effective cloud analytical database as a service to be used as centralized data warehouse for company of any size.

Re: Show HN: Open-Source Business Intelligence for BigQuery – Looker Alternative

#45
post #40

Earlier quoted context omitted.

@akalitenya the way I used to explain it to the technical audience is: would you rather program in a low-level language like C or in a high-level language like Java, Python, or Node? Well, if you are writing compilers, you might choose C, but if you are doing almost everything else, you want to choose a language that fits the purpose: python for data science, Node for web backend, etc. Same with LookML - it provides…

@segah Good point. Based on your consulting experience, can you tell if there have been any cases when your clients chose another Business Intelligence tool instead of Looker, what are these tools and what are the reasons?

that's the wrong question. People choose different tools for all kinds of reasons. And many of these reasons are totally valid. A tool, Looker or otherwise, is no panacea.

Re: Show HN: Open-Source Business Intelligence for BigQuery – Looker Alternative

#46
post #7

@akalitenya are you even in the clear with this? Some of the Old LookML syntax is an exact copy. But more importantly, the challenge for any such tool is to go beyond use by 2-3 people. At 2-3 people anything will work. Where BI tools (open source and close source) struggle is scale: having all the right features for, essentially, a group of users who actually don't know how to work with data (did I just say that alo…

Hey @segah, founder of Chartio here. I can deeply second all the comments on how long of a feature tail BI is, and the amount of work it takes to have real product depth and stability vs. an impressive demo. It takes years, and ongoing maintenance that is impossible to estimate in the beginning.

Also, it's definitely a challenge to support 100's and 1000's of users digesting data, especially in the democratized fashion that we're typically used in. It takes good data governance, support, and admin tools. I gotta chime in and say for the record though that Chartio well supports many such customers.

It may be weird for you to chat as you worked at Looker, but I'd love to hear anytime on why you see Chartio capping out at a lower # than the others listed. You can reach me at dave-at-chartio.com!

Post reply on HN