Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What does your BI stack look like?

news.ycombinator.com

Post reply on HN
Print view View original

101–110 of 181 posts

Re: Ask HN: What does your BI stack look like?

#101

Post by ska »

nm2259 wrote: post #72

Data infrastructure is scattered, siloed, excel sheets and google sheets stored in various places (personal g drives, company g drives, some network share somewhere, an ms sharepoint site, sometimes in development git repos or various wikis) Reporting infrastructure is manual massaging and lots of powerpoint. My company isn't that large, but bigger than you'd think for such a "system".

> but bigger than you'd think for such a "system".

I suspect you underestimate how large some of the organizations are that use a variant of this system.

Re: Ask HN: What does your BI stack look like?

#102

Post by perlgeek »

nm2259 wrote: post #72

Data infrastructure is scattered, siloed, excel sheets and google sheets stored in various places (personal g drives, company g drives, some network share somewhere, an ms sharepoint site, sometimes in development git repos or various wikis) Reporting infrastructure is manual massaging and lots of powerpoint. My company isn't that large, but bigger than you'd think for such a "system".

That's the reality at many organizations, even for 100+ employee companies.

Re: Ask HN: What does your BI stack look like?

#103

Post by netcraft »

rajivayyangar wrote: post #82

As a data scientist (startup / Yahoo) turned product manager (4 startups), I've used a variety of stacks in the past - from plain SQL, to Mode, to Mixpanel, Looker, Interana, and Hive. Recently we started using PopSQL ( https://popsql.com ) and love it. When I don't have a dedicated data team, my philosophy is: 1) Make it difficult to get wrong answers - Don't use Google Analytics. It's too easy to generate incorrect…

$20/user/month seems incredibly expensive for an editor that we would have traditionally paid a one time cost for. I assume this has the saving and sharing of queries built in, but that is a lot of money for some storage. Are there other major benefits I am not seeing? (It is not my intention to attack you or make you defend this product, I am just curious if I am missing something - there are a lot of similar products with similarly questionable value propositions)

Re: Ask HN: What does your BI stack look like?

#104

Post by chrisjc »

numlocked wrote: post #76
ericmjalbert wrote: post #70

Earlier quoted context omitted.

Just curious, if you could highlight some of the problems you've experienced with each "generation" of looker reporting? Was there any specific issues that encouraged the re-implementation?

Yep — problems with the first gen were: 1. Complex joins and poor performance. 2. Not pre-computing important attributes made them hard to slice and dice on (because they weren’t true dimensions) 2nd gen: 1. Really unmaintainable. Like.... really unmaintainable. A rat’s next of ad-hoc common table expressions, that referred to each other in undocumented ways, used inconsistent CASE statements to transform enums to hu…

> If we had been smarter, we would have introduced DBT after gen 1 and not gotten into the gen 2 mess.

Is this a roundabout way of saying you wouldn't have exposed looker to the underlying staged data?

We have two layers within our data warehouse: The ingestion/staging layer, and the organized/cleansed/segmented layer. We only give Looker projects access to the latter.

We still have a lot of chaos within our Looker projects (views with select star, dimensions for every column bc of select star, hard-coded schemas, etc...). Slowly working it out though.

Re: Ask HN: What does your BI stack look like?

#105

Post by chrisjc »

pcarolan wrote: post #10
numlocked wrote: post #9

At Grove, we are: Airflow -> S3 -> DBT with Spark/EMR or Redshift/Spectrum -> Redshift data marts -> Looker At least, that’s the way we like our pipelines to work. In practice we have a couple of extractions that land directly in Redshift (we extract Zendesk data, for instance, with Stitch Data). We use Snowplow for click stream analytics. And we’ll likely move from Redshift to Snowflake (or mayybbbeee Bigtable) in Q…

Very similar to our progression, do you have a good way you manage your data dictionary and catalog?

Not a complete solution, but if you're already using Looker...

https://looker.com/platform/blocks/embedded/create-a-data-di...

Re: Ask HN: What does your BI stack look like?

#106

Post by rmk2 »

We are a small data consultancy, so we use other/more diverse things for customers, but our internal stack is fairly simple:

Hubspot/Jira/G Suite → (Python) → PostgreSQL → (SQL) → Tableau

Since we are Tableau partners, we have a bunch of internal licences either way. We host Tableau Server, ETL, and PostgreSQL ourselves, all on Centos cloud servers.

Re: Ask HN: What does your BI stack look like?

#107

Post by chrisjc »

vitorbaptistaa wrote: post #16

Luigi, AWS S3, DBT, Snowflake and Re:dash (currently analyzing Metabase or Looker to allow queries without SQL) Luigi runs our scrapers and other workflow management tasks (e.g. DB backups). All raw data lives in S3. We make an effort to be able to recreate the whole data warehouse from the raw data, so if any cleaning/normalization process fails, we have this safety net. I'm curious to hear if others use a similar p…

> All raw data lives in S3... I'm curious to hear if others use a similar pattern, or if there are better options.

You still may need to maintain information about what is in S3. If you're already using Snowflake you can probably achieve this via External tables. Otherwise you could run your own hive meta-store or use AWS external tables.

We're also trying to decide whether it's cheaper/easier to store things in S3 or just keep everything internal in snowflake. In some cases the compression is so good on some of our data, it's better keeping data stored in a table than in internal or external staging. Obviously this is bad if we ever have to move away from Snowflake, but we haven't committed to either approach and lots of data is still backed up in S3. Our total data warehouse is about 200TB at the moment and we're projecting significant growth over the next couple of years.

Re: Ask HN: What does your BI stack look like?

#108

Post by chrisjc »

mmckelvy wrote: post #63

A lot of these answers seem to focus on app analytics (e.g. collecting clicks, page views, etc. from Segment). How are people collecting / integrating financial data (e.g. sales, subscriptions, expenses)?

So how are you collecting/integrating financial such as what is contained in salesforce/netsuite/etc?

My recommendation would be

- an ETL as a service such as fivetran, stitch, etc

- into a data warehouse such as snowflake, big-query, etc

- transform/aggregate/normalize/magic data using some ETL tool such as dbt, matillion, etc

- analytics tool such as Looker, Tableau, etc...

Re: Ask HN: What does your BI stack look like?

#109

Post by chrisjc »

EdwardDiego wrote: post #98

Kafka streaming to validate and attach model data, fed into Druid by Flink, queried via a custom front end.

> and attach model data

Can you elaborate on this? Do you mean enriching events and/or windowing/aggregating/summarizing?

Are you using KSQL?

Re: Ask HN: What does your BI stack look like?

#110

Post by hundredwatt »

seveibar wrote: post #13

Postgres -> Metabase I believe this is the best combination of cheap/powerful for early-stage startups. My very non-technical cofounder is able to use metabase's simple GUI interface to create graphs/insights (even joining and aggregating across tables!), and for anything complex I can step in a give a helper SQL query. We have around 10M records we aggregate around for daily insights. Metabase can also be run as a p…

+1 for Metabase

For our team, using an ELT architecture (as opposed to ETL) [1] for managing our data warehouse has greatly reduced the complexity of our data processes. Instead of creating ETLs for every table we want to load into the data warehouse, we create the minimum necessary setup to copy the table into our data warehouse. Then, we write transforms, which are simply SQL statements, to generate wide-column tables that our non-technical users can use to explore data without worrying about joins or having to learn esoteric naming conventions.

Custom EL Scripts -> Redshift -> Transform Statements -> Redshift -> Metabase supports the data needs of all our departments with no dedicated data team members.

[1] https://www.dataliftoff.com/elt-with-amazon-redshift-an-over...

Post reply on HN