41–50 of 181 posts
BI?
The addition of Dataform into our stack has completely changed the game, allowing us to maintain one source of objective truth and maximise looker (minimising the joining etc. in LookML, instead pointing it straight at tables in BigQuery).
This in turn means our downstream data users are happier to use a fast and reliable looker instance!
If you have a data team: Stitch / Segment -> BigQuery -> Dataform -> BigQuery -> Looker I work with many companies helping them set up their data stack, and from what I've seen this is pretty much the optimal set up. These tools all require very little maintenance, are relatively cheap (compared to the man power required to set these things up from scratch internally), and scale well as companies grow and complexity…
Without design sessions to figure out your data store design (look up Kimball, Immon), and then monitoring/testing to make sure everything is running smoothly, any data stack will be hell.
Badly designed data brings fatigue and rancor and unnecessary work and is a huge hard-to-quantify money suck.
BI?
"Two words combined that won't make sense" etc.
Everyhting goes through S3 because Snowflake storage is on it.
dbt is amazing, we began using it a month ago and it already transformed the way our data team work. It really is a value multiplier for everyone. Data engineers are happier because they don't need to write and maintain data transformations, analysts are happier because they can maintain their own SQL pipelines & the whole company is happier because we now have a great documentation tool to explore our data.
We also are big fans of Snowflake, make operating a data warehouse a breeze.
Then, we use a mix of Redash & Tableau for reporting.
Redash for static reporting (open to the whole company) & Tableau to create more complex data tools we expose to some internal teams; Marketing, Risk, Finance ...
Some spark mixed in at the ingestion and transformation phases.
Like someone else said in this thread, we're currently battling Redshift scaling problems and are looking to offload more of the raw data to S3 using Spark to create read views.
No data catalog right now but the Looker setup is extremely well-maintained. Hoping to deploy Lyft's Amundsen to fix this more generally.
We basicallly don't have a BI stack. Which is silly, but I guess we've had other priorities.
MySQL -> Data export using Sqoop through Airflow -> S3 -> Spark -> Jupyter Notebook
PS: Qubole is a data platform which makes ETL pipeline setup easy.