Infra: Postgres -> S3 -> Looker
Business: Segment -> customer.io/Zapier/Heap Analytics + Looker
Support/Success: Intercom+Slack -> frame.ai
Ask HN: What does your BI stack look like?
71–80 of 181 posts
Re: Ask HN: What does your BI stack look like?
#72Data infrastructure is scattered, siloed, excel sheets and google sheets stored in various places (personal g drives, company g drives, some network share somewhere, an ms sharepoint site, sometimes in development git repos or various wikis)
Reporting infrastructure is manual massaging and lots of powerpoint.
My company isn't that large, but bigger than you'd think for such a "system".
Re: Ask HN: What does your BI stack look like?
#73We're a small, bootstrapped company with 2 people. Some data is millions of rows others are billions.
Goal for us is KISS. Keeping everything as simple as possible -- both in terms of infrastructure, ease of use and cost.
Primary we're using Metabase in front of multiple MariaDB instances. Metabase is absolutely fantastic!
We also have a couple of additional small projects:
- A small Node.js app that receives events live via fluentbit, saves them off in MariaDB and sends text message notifications via Twilio when certain thresholds are exceeded
- A small "ETL" app that polls the Stripe and QuickBooks API to mirror data in a MariaDB database so we can easily access it from Metabase
Re: Ask HN: What does your BI stack look like?
#74We keep it very simple as well.
Postgres read-replica for cheap realtime stuff, Redshift for the larger sets, Airflow to manage transfers, and Metabase to visualize and share. We also collect data from a bunch of APIs, but those each run via their own job and store in Postgres.
We also try to define wide short tables in Redshift that enable users to use Metabase to ask questions.
I was very happy with Metabase. Being that we can't afford Looker right now (but we would all love to) it is pretty solid.
Re: Ask HN: What does your BI stack look like?
#75We use BigQuery as the data warehouse - there are airflow jobs to periodically load data into BQ - most of these are simple python scripts.
Metabase for most of our simple BI metrics - Tableau for some advanced users doing more complicated stuff.
Re: Ask HN: What does your BI stack look like?
#76At Grove, we are: Airflow -> S3 -> DBT with Spark/EMR or Redshift/Spectrum -> Redshift data marts -> Looker At least, that’s the way we like our pipelines to work. In practice we have a couple of extractions that land directly in Redshift (we extract Zendesk data, for instance, with Stitch Data). We use Snowplow for click stream analytics. And we’ll likely move from Redshift to Snowflake (or mayybbbeee Bigtable) in Q…
Just curious, if you could highlight some of the problems you've experienced with each "generation" of looker reporting? Was there any specific issues that encouraged the re-implementation?
Yep — problems with the first gen were:
1. Complex joins and poor performance. 2. Not pre-computing important attributes made them hard to slice and dice on (because they weren’t true dimensions)
2nd gen:
1. Really unmaintainable. Like....really unmaintainable. A rat’s next of ad-hoc common table expressions, that referred to each other in undocumented ways, used inconsistent CASE statements to transform enums to human-readable values, inconsistent time stamp formats (and time zones), etc, etc, etc. Fields in different explores that were named the same but didn’t quite match and no one could explain why. 2. All of it was SQL in LookML and the persistence strategy (e.g. when were the CTEs running, were they writing to disk or ephemeral, etc) was not manageable.
If we had been smarter, we would have introduced DBT after gen 1 and not gotten into the gen 2 mess.
There’s a ton of other nuance here, but that’s the high level.
Re: Ask HN: What does your BI stack look like?
#77Postgres -> Metabase I believe this is the best combination of cheap/powerful for early-stage startups. My very non-technical cofounder is able to use metabase's simple GUI interface to create graphs/insights (even joining and aggregating across tables!), and for anything complex I can step in a give a helper SQL query. We have around 10M records we aggregate around for daily insights. Metabase can also be run as a p…
UX guy from Metabase here. Glad to hear it's working well for you. Happy to answer any questions folks might have.
Re: Ask HN: What does your BI stack look like?
#78Luigi, AWS S3, DBT, Snowflake and Re:dash (currently analyzing Metabase or Looker to allow queries without SQL) Luigi runs our scrapers and other workflow management tasks (e.g. DB backups). All raw data lives in S3. We make an effort to be able to recreate the whole data warehouse from the raw data, so if any cleaning/normalization process fails, we have this safety net. I'm curious to hear if others use a similar p…
Let me know if you have any questions so far about Metabase (I'm on the core team).
Re: Ask HN: What does your BI stack look like?
#79@sixo @numlocked I'd be happy to share more about Snowflake's
architecture. As others mentioned in this thread, Snowflake completely separates storage from compute -- eliminating Redshift scaling issues.
Drop me a message, would love to chat.
william.ladow@snowflake.com
Re: Ask HN: What does your BI stack look like?
#80Earlier quoted context omitted.
Can you share details on how much operating that BI SaaS stack costs you monthly?
We’ve been battling Redshift scaling problems so it’s fluctuated a lot over the last 6 months (both up and down). $15k per month to an order of magnitude (not counting Looker licensing).
That seems like a lot. Snowflake typically cuts Redshift bills in half if you are hitting limits with RS