Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What does your BI stack look like?

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 181 posts

Re: Ask HN: What does your BI stack look like?

#51

Post by buremba »

sixo wrote: post #48

Custom pipes mostly coordinated by Luigi (heavily customized) -> Redshift -> DBT -> Looker. Some spark mixed in at the ingestion and transformation phases. Like someone else said in this thread, we're currently battling Redshift scaling problems and are looking to offload more of the raw data to S3 using Spark to create read views. No data catalog right now but the Looker setup is extremely well-maintained. Hoping to…

Since your Looker setup is well-maintained, did you consider adopting the data catalog in Looker, what would be the challenge doing it?

Re: Ask HN: What does your BI stack look like?

#53

Post by grahamdietz »

We use Snowplow directly into Snowflake and report on this using Mode. We are a data native SaaS firm, and we set things up like this about 5 years ago and it has served us well. Streamlined and high performance. For all other sources, we use Stitch -> Snowflake, with one semi-custom Singer.io tap also running on Stitch. All this was simple to set up and means we don't have to worry about getting the data into one place. Of course, we then have lots of work to do in Snowflake and Mode to get the data the business needs. We share the reports from Mode to various teas via Slack. Hope this helps.

Re: Ask HN: What does your BI stack look like?

#54

Post by aaronharnly »

At Amplify, Matillion/Stitch/Fivetran/Custom -> S3 + Snowflake -> Matillion for transforms -> Looker + Tableau.

It's important to say out loud that a lot of analysis also happens within product-local reporting systems, or in "offline" Google sheets.

We are currently working on building out the same stack, terminating in a Powered By Looker instance, for customer-facing aggregate reporting.

The engineering and data science teams do great with Looker, but Tableau still covers use cases for non-engineer business people (think: the finance, customer operations, training, etc) who want to start from source data plus a few sidebar Google sheets, tinker around, and then develop a rough report without waiting for iterative cycles with the data engineering team. We're thinking hard about how to get those use cases into the warehouse + Looker faster.

Re: Ask HN: What does your BI stack look like?

#55

Post by mharroun »

Ive done 2 data pipelines, one alot like what most people are are talking.

The other I had to build for a startup with millions of monthly uniques but only seed funding (cant do a 30+k a month data eng bill).

Went with custom event emission->kenisis->(druid & S3) and used imply (https://imply.io/). Easy real time data analytics, auto "enrichment with druid lookups from a RDBMS, and a simple ui for slice/dice investigation metrics. All in all costed lest then the cheapest looker license.

Re: Ask HN: What does your BI stack look like?

#56

Post by sejtnjir »

Informatica -> ADLS -> SQL Server -> PowerBI also, in the same department: NiFi -> HDFS -> Spark -> Hive -> NiFi -> ADLS -> PowerBI and: NiFi -> Azure Event Hubs -> Azure container instances -> Event Hubs -> Streaming Analytics -> PowerBI

I'm pretty fond of the last stack for streaming dashboards in the sensor data realm.

Re: Ask HN: What does your BI stack look like?

#57

Post by conradfr »

bass_case wrote: post #52

Are any of you offering embedded analytics in your products? Mostly applies to mid-market/enterprise SaaS platforms but interested in learning more about how you offer analytics/BI to your customers in your products.

I'm not on the data team but on the dev side and we are currently looking to do that with Looker.

After looking at the different way of doing that with their API I think we'll have to use that https://docs.looker.com/reference/embedding/sso-embed for a mix of convenience and security.

(I'm not affiliated with Looker and have no opinion about it ... yet).

Re: Ask HN: What does your BI stack look like?

#58

Post by wrs »

Various -> Airflow -> BigQuery -> Looker

We have a variety of data sources, from Mixpanel to PostgreSQL to black-box systems that can only dump out CSV files. We use Airflow to manage pulling from those, doing some transforms, and writing everything into BigQuery. Looker then handles the reporting and browsing.

Re: Ask HN: What does your BI stack look like?

#59

Post by aceregen »

bass_case wrote: post #52

Are any of you offering embedded analytics in your products? Mostly applies to mid-market/enterprise SaaS platforms but interested in learning more about how you offer analytics/BI to your customers in your products.

Hi! You may like to check out what our team has built at www.holistics.io. The pricing metrics of our embedded dashboards cater for you to scale to unlimited viewers, and you don't need to recreate a duplicate copy of your user accounts into our system for it to work. Check out the video below to find out more!

https://www.holistics.io/guides/embed-analytics/

Re: Ask HN: What does your BI stack look like?

#60

Post by sixo »

buremba wrote: post #51
sixo wrote: post #48

Custom pipes mostly coordinated by Luigi (heavily customized) -> Redshift -> DBT -> Looker. Some spark mixed in at the ingestion and transformation phases. Like someone else said in this thread, we're currently battling Redshift scaling problems and are looking to offload more of the raw data to S3 using Spark to create read views. No data catalog right now but the Looker setup is extremely well-maintained. Hoping to…

Since your Looker setup is well-maintained, did you consider adopting the data catalog in Looker, what would be the challenge doing it?

The main reason is that only a subset of all our datasets are actually exposed in Looker, and Amundsen would have applicability to other data sources - certain production semi-OLAP DBs, the future hive-on-S3 deployment... that we have no plans to expose in Looker at all.

That said I haven't looked closely at how far we could go with Looker, I'll check it out.

Post reply on HN