Ask HN: How to go about the goal of democratising data at our organisation • Hacker News Board

Post by tarungarg546 » Tue, Sep 10, 2019, 3:53 PM UTC

Hi Everyone! Hope you all are doing great.

We(@ squadvoice.co) wanted to take your help/suggestion in something we're trying and figuring out if to build or not. Below is the problem statement that we're focusing on majorly. Become a data-driven organization where anyone can make decisions & leverage data in a self-service model without spending too much time on data retrieval or data query formation. Why we're thinking we should build a solution for the same?

1) Ad-hoc analysis requires engineering bandwidth at a lot of time. And due to that product features get delayed.

2) New queries require engineering support at a lot of times.

3) Operational queries become stale (inconsistent) with changing business logic.

4) Queries timeout, run slow, etc.

5) We take lots of decision basis the data, and if data fetching and gathering is slow and time taking it is gonna slow whole org down.

We do have SQL dashboarding and things in place, but new requirements keep coming up(I guess we're trying to find new data points every day to see what we can improve which is great honestly) and application+business logic change so often that it is hard to keep it updated.

Now, we wanted to understand:-

1) How people in other organizations like yours are solving the same?

2) What are the things you tried what works and what does not?

3) How do you validate whether it is something that you should solve for or not?

4) What're technologies and gotchas that you think should we be aware of and consider for this kind of things?

5) How do you make sure, whatever you build is scalable and alterable such that if you change your application logic in future things still remains sane.

6) How do you make sure, people are able to get data they need as fast as they can.

I am already sorry if this post looks long and some redundant elements, but I wanted to communicate problem and what we're looking for help in as clear as possible.

Post by seektable » Thu, Sep 12, 2019, 8:28 AM UTC

First step is providing simple self-service access to data with minimal time/money efforts. Typical approach is:

* data sources (database tables & columns) are isolated from end users with some kind of 'model'; end-users operate in terms of this model. This model is created & maintained by 'publishers' - this is user(s) with some technical background that know database structure & SQL good enough. In large organizations this is IT specialists, in small businesses this role is covered by founders/managers with tech skills. If organization database(s) cannot handle OLAP queries directly some kind of data lake is organized to deal with slow queries.

* end users use BI tool to perform ad-hoc analysis, save & share their reports.

Also it is good idea to start from something very simple that can be configured very fast; in this way you can collect feedback from end-users (BI consumers) as early as possible, and then decide if you need to invest more in more complex / enterprise oriented BI solutions. In this context, I may recommend to check our BI tool: https://www.seektable.com (both cloud service and on-premise installations are possible).