How does your company visualize and report on what’s happening in the business?
Ask HN: What does your BI stack look like?
1–10 of 181 posts
Re: Ask HN: What does your BI stack look like?
#2Formerly: Oracle Hyperion -> ETL to Azure -> PowerBI
Currently: Internal Data Warehouse -> RDS -> Internal web app (Django, React)
Re: Ask HN: What does your BI stack look like?
#3{SQL Server, Oracle} -> ETL to SQL Server (Hybrid graph db)-> SQL Server Analysis Services -> {Power Bi, Tableau}
Re: Ask HN: What does your BI stack look like?
#4Looker on top of Redshift. Events streamed in from Segment and ELT transforms managed by Airflow. Looker gives you nice visualizations and charting with RBAC and some some lightweight ETL functionality. The real advantage of Looker is their modeling layer which sits on top of your physical data and is done via a DSL called LookML. Source control is managed via a wrapper around git. The end result is that analysts can move lightning fast building their own models in SQL or do analysis via drag and drop explorations. Looker's customer support is the best I've experienced and hasn't changed since Google acquired Looker. We're likely moving off Redshift to Snowflake in the next 6 months because it is slow to pick up changes and we want to decouple storage and compute for scaling reasons. Airflow is an incredible platform but needs customization and templated workflows to make it great. Data build tool (DBT) is how we plan on managing ETL code via Airflow in the near future. We're also adding Spark, but more for data science.
Re: Ask HN: What does your BI stack look like?
#5Daily ETL in form of sql scripts from prod dbs to internal
data warehouse for use via Saiku, case-by-base Shiny apps
and some bash email-reporting.
Re: Ask HN: What does your BI stack look like?
#6apps on pivotal to extract, transform, combine data -> to ms sql db -> powerbi to visualize
Re: Ask HN: What does your BI stack look like?
#7[deleted]
Re: Ask HN: What does your BI stack look like?
#8Data warehouse:
- Amazon Redshift (data sourced from many different backend DBs; e.g. PostgreSQL, MySQL, other Redshift instances etc.)
- BigQuery (Some teams store the data for their projects here. For reporting, they're selectively transferred to Redshift.)
Reports:
- Tableau (extracts of pre-aggregated data from Redshift)
- Looker (connects live to and executes its own queries on Redshift)
Anything that is based on pre-aggregated (rather small, e.g. n_rows ETLs for raw data run daily on Redshift. Reports are also scheduled on Redshift (with Airflow) but the refresh interval is report-specific.
Re: Ask HN: What does your BI stack look like?
#9At Grove, we are:
Airflow -> S3 -> DBT with Spark/EMR or Redshift/Spectrum -> Redshift data marts -> Looker
At least, that’s the way we like our pipelines to work. In practice we have a couple of extractions that land directly in Redshift (we extract Zendesk data, for instance, with Stitch Data). We use Snowplow for click stream analytics. And we’ll likely move from Redshift to Snowflake (or mayybbbeee Bigtable) in Q2 of 2020.
We used to do all of our transforms via DBT within Redshift but have been offloading the heavier-duty pieces (like Snowplow event processing) to Spark jobs because they were too taxing on Redshift.
We’ve gone through 3 “generations” of Looker reporting — gen 1 was just wrapping LookML around our schema and forcing Looker to do the joins and generate SQL for everything we wanted to know. Second generation involved a bunch of common table expressions within Looker itself that worked, but were developed without much thought as to data mart design. Gen 3 is where we are now with super deliberate scoping and implementation of warehouses in DBT. Before any of that we rolled our own tool [0].
Happy to answer any questions. We’ve gone from ~30 to ~1000 employees in the 3 years since we started using Looker and have learned a thing or two along the way.
Re: Ask HN: What does your BI stack look like?
#10At Grove, we are: Airflow -> S3 -> DBT with Spark/EMR or Redshift/Spectrum -> Redshift data marts -> Looker At least, that’s the way we like our pipelines to work. In practice we have a couple of extractions that land directly in Redshift (we extract Zendesk data, for instance, with Stitch Data). We use Snowplow for click stream analytics. And we’ll likely move from Redshift to Snowflake (or mayybbbeee Bigtable) in Q…
Very similar to our progression, do you have a good way you manage your data dictionary and catalog?