Ask HN: What does your BI stack look like?

We use Segment for event tracking, Postgresql for transactional data and a number of spreadsheets and third-party integrations with Stitchdata. Since our data is relatively small, we use PG as a data-warehouse and heavily use DBT for ETL. The people who are familiar with SQL just use DataGrip, for the UI we use our tool https://rakam.io.

Shameless plug: It's basically similar to Looker but it plays well with the product data and integration with Segment as well.

No :)

We are revamping our management of Snowplow event structures presently to make them more semantic and easier to use.

For our core data marts, we rely on naming conventions for the LookML fields, good tool tips, hold monthly training sessions for new employees, weekly office hours with analysts, and do occasional “train the trainer” sessions to try and make sure there is at least one Looker power-user in each business team.

Luigi, AWS S3, DBT, Snowflake and Re:dash (currently analyzing Metabase or Looker to allow queries without SQL)

Luigi runs our scrapers and other workflow management tasks (e.g. DB backups).

All raw data lives in S3. We make an effort to be able to recreate the whole data warehouse from the raw data, so if any cleaning/normalization process fails, we have this safety net. I'm curious to hear if others use a similar pattern, or if there are better options.

DBT handles both loading the data from S3 into Snowflake (by creating the Snowflake Stages), and transforming the data in Snowflake. This isn't how DBT is used usually, but it felt wasteful to add Stitch or another tool to load the data into Snowflake, as snowflake supports it out of the box. I also created a `setup_snowflake` macro operation that creates our users, warehouses, databases, etc., in Snowflake (a kind of "poor man's Terraform")

I don't think Snowflake requires introduction. It's an amazing tool. We used Postgres before, but Snowflake is much much better, even though our DB is pretty small (~200 GB).

Finally, we use Re:dash as a BI, but I'm checking other options that allow usage without SQL (currently Metabase and Looker).

I don't want to derail the thread, but I recently started a company (https://syndetic.co) that's working on this problem. We've been focusing on the external data dictionary use case (how does a data-as-a-service company explain a dataset to their prospective customers) but we've been encountering a number of companies that are evaluating data catalogs and other internal tools for their data science teams.

I would really appreciate getting your perspective - I'm steve (at) syndetic.co

Stitch -> Redshift (with DBT for data modeling) -> Looker

For a smaller company, it makes a lot of sense for us to use off-the-shelf tools like these rather than rolling our own pipelines. Every once in a while we run into issues where Stitch isn't quite as flexible as we'd like (can't set dist/sortkeys etc), but that's the tradeoff you make for its simplicity.

DBT is amazing and I can't recommend it highly enough!

Looker works for analytics, but we're starting to do more data-sciency work, and it doesn't have built-in Jupyter notebooks or anything like that. Does anyone have a solution they use and like for hosting and sharing notebooks alongside a stack like this?