Ask HN: What does your BI stack look like? • Hacker News Board

Re: Ask HN: What does your BI stack look like?

#141

Post by thenaturalist » Tue, Nov 12, 2019, 11:24 PM UTC

mike_lee_28 wrote: post #43

We've recently changed from : Segment -> BigQuery -> Looker to Segment -> BigQuery -> Dataform -> BigQuery -> Looker. The addition of Dataform into our stack has completely changed the game, allowing us to maintain one source of objective truth and maximise looker (minimising the joining etc. in LookML, instead pointing it straight at tables in BigQuery). This in turn means our downstream data users are happier to us…

Your first comment ever?

Any affiliation with DataForm?

Re: Ask HN: What does your BI stack look like?

#142

Post by EdwardDiego » Tue, Nov 12, 2019, 11:28 PM UTC

chrisjc wrote: post #109

EdwardDiego wrote: post #98

Kafka streaming to validate and attach model data, fed into Druid by Flink, queried via a custom front end.

> and attach model data Can you elaborate on this? Do you mean enriching events and/or windowing/aggregating/summarizing? Are you using KSQL?

Yeah, enriching the events as they pass through - so we have a Kafka Connect JDBC source polling the source of truth DB every X seconds for model changes (we looked into Debezium for streaming model changes, but the additional complexity didn't bring us much benefit based on how often our model changes), and then we pull that into a GlobalKTable and join the KStream as appropriate.

We also use Kafka streaming with a persisted event window store to deduplicate across a window of 2 hours.

Re: Ask HN: What does your BI stack look like?

#143

Post by eibhinn » Tue, Nov 12, 2019, 11:35 PM UTC

Oracle -> IBM Cognos

A nightly rebuild using ETL scripts written in sql. Not cheap or glamorous, but solid for our needs.

Re: Ask HN: What does your BI stack look like?

#144

Post by spullara » Wed, Nov 13, 2019, 1:02 AM UTC

Kinesis Firehose -> S3 -> Snowflake -> Sigma

Re: Ask HN: What does your BI stack look like?

#145

Post by exabrial » Wed, Nov 13, 2019, 1:03 AM UTC

MySQL -> replication -> MySQL -> Metabase & Tableau

We want to switch to postgres because of features, but "The devil you know is better than the devil you don't", so we just kinda sticking with MySQL.

Re: Ask HN: What does your BI stack look like?

#146

Post by exabrial » Wed, Nov 13, 2019, 1:08 AM UTC

mazameli wrote: post #77

seveibar wrote: post #13

Postgres -> Metabase I believe this is the best combination of cheap/powerful for early-stage startups. My very non-technical cofounder is able to use metabase's simple GUI interface to create graphs/insights (even joining and aggregating across tables!), and for anything complex I can step in a give a helper SQL query. We have around 10M records we aggregate around for daily insights. Metabase can also be run as a p…

UX guy from Metabase here. Glad to hear it's working well for you. Happy to answer any questions folks might have.

Great Product. And you guys are great stewards on Github.

Re: Ask HN: What does your BI stack look like?

#147

Post by kmerrol » Wed, Nov 13, 2019, 1:25 AM UTC

A bit different from the crowd here. After much searching, our BI stack has been fully converted to data streaming using: StreamSets DataCollector >> Kafka >> AWS S3 >> AWS Redshift >> Dremio >> Jupyter Notebooks. Great to have Jupyter take on data prep and data analysis tasks while external tables in Redshift are very fast with minimal ETL. Dremio has been great as a virtual data mart with great performance on Parquet data in S3 and desktop BI tool integration.

Re: Ask HN: What does your BI stack look like?

#148

Post by masonhensley » Wed, Nov 13, 2019, 2:49 AM UTC

apinstein wrote: post #130

seveibar wrote: post #13

Postgres -> Metabase I believe this is the best combination of cheap/powerful for early-stage startups. My very non-technical cofounder is able to use metabase's simple GUI interface to create graphs/insights (even joining and aggregating across tables!), and for anything complex I can step in a give a helper SQL query. We have around 10M records we aggregate around for daily insights. Metabase can also be run as a p…

+1 for metabase. I’ve deployed it at 2 companies. The latest version allows you to compose “questions” you create in the system which is really powerful for organizing layers of data reporting. Also, their API conforms 100% to Zapier’s polling feed structure, and I managed to build an integration that syncs our customer base to Pipedrive in a few hours. A bit more time and I have it automatically moving deals along i…

Interesting, kinda like this? https://blog.synq.fm/how-we-optimized-our-crm-workflow-using...

P.S. I'm another Metabase + Postgres + Bigquery fan. Thanks Team!

Re: Ask HN: What does your BI stack look like?

#149

Post by subhajeet2107 » Wed, Nov 13, 2019, 4:00 AM UTC

Custom Collector(analytics) -> Clickhouse -> Custom ETL Scripts -> Clickhouse -> Re:dash We tried metabase which is awesome , but Redash is also great and easy to setup as well if your team knows sql then Redash is better We also looked at druid and after some benchmarking we settled on Clickhouse, realtime queries even without etl runs within seconds in clickhouse

Re: Ask HN: What does your BI stack look like?

#150

Post by wilbo » Wed, Nov 13, 2019, 5:32 AM UTC

Postgresql > pgadmin

For less technical people > metabase

For automated reporting and storing historic trends > Klipfolio

For near real time automated operational reporting > kloud.io