Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What does your BI stack look like?

news.ycombinator.com

Post reply on HN
Print view View original

91–100 of 181 posts

Re: Ask HN: What does your BI stack look like?

#91

Post by buremba »

sixo wrote: post #60
buremba wrote: post #51

Earlier quoted context omitted.

Since your Looker setup is well-maintained, did you consider adopting the data catalog in Looker, what would be the challenge doing it?

The main reason is that only a subset of all our datasets are actually exposed in Looker, and Amundsen would have applicability to other data sources - certain production semi-OLAP DBs, the future hive-on-S3 deployment... that we have no plans to expose in Looker at all. That said I haven't looked closely at how far we could go with Looker, I'll check it out.

I see. We actually develop a BI tool similar to Looker but also want to implement a data catalog feature embedded into it. I'm familiar with Amundsen but I believe that this kind of tool needs to have tight integration with the BI tool and Amundsen still has a way in terms of integrations. (searching among saved reports, dashboards and collaboration with teams, etc.). Otherwise, using spreadsheets is still easier and maintainable.

I would love to hear the features that you would need in a data catalog system.

Re: Ask HN: What does your BI stack look like?

#95

Post by pcarolan »

seveibar wrote: post #13

Postgres -> Metabase I believe this is the best combination of cheap/powerful for early-stage startups. My very non-technical cofounder is able to use metabase's simple GUI interface to create graphs/insights (even joining and aggregating across tables!), and for anything complex I can step in a give a helper SQL query. We have around 10M records we aggregate around for daily insights. Metabase can also be run as a p…

Agreed. I helped setup a company on this last year and they're very happy. There's a 'Deploy to Heroku' push button install that was incredibly easy to setup. Great job Metabase folks.: https://www.metabase.com/start/heroku

Re: Ask HN: What does your BI stack look like?

#96

Post by pcarolan »

stevepike wrote: post #18
pcarolan wrote: post #10

Earlier quoted context omitted.

Very similar to our progression, do you have a good way you manage your data dictionary and catalog?

I don't want to derail the thread, but I recently started a company ( https://syndetic.co ) that's working on this problem. We've been focusing on the external data dictionary use case (how does a data-as-a-service company explain a dataset to their prospective customers) but we've been encountering a number of companies that are evaluating data catalogs and other internal tools for their data science teams. I would…

Was just thinking this would make a great business opportunity. Thanks, we'll check it out!

Re: Ask HN: What does your BI stack look like?

#99

Post by perlgeek »

EdwardDiego wrote: post #98

Kafka streaming to validate and attach model data, fed into Druid by Flink, queried via a custom front end.

We use something similar in the first half, Kafka -> Flink -> (Druid | PostgreSQL) -> Metabase

We use Druid for time series data, and Postgres for everything else.

Re: Ask HN: What does your BI stack look like?

#100

Post by chrisjc » 

    -> Kafka-connect -> Snowflake -> SQL/sf-tasks -> Snowflake -> Looker
    -> Alooma        ->
    -> custom        ->
Using Kafka-connect, we're able to serve up near real-time (2-5 mins) insights on device generated events.

We probably need to use some kind of ETL tool to replace custom SQL and sf-tasks. Unfortunately, we haven't been able to find a tool that handles this in a non-batch (even if it's micro-batching) form. Snowflake change-streams and tasks allows us to ETL in a streaming-like fashion.

We're ingesting everything from raw/transformed/aggregated events, micro-service DBs (as fast as they sprout up), netsuite/salesforce, mixpanel, MySQL, MongoDB... Billions of rows of data across multiple data-source accessible to internal and external customer in a matter of seconds. It's been an incredible challenge, especially with only a team of 2-5 people.

Post reply on HN