Ask HN: Lessons learned from implementing user-facing analytics / dashboards? • Hacker News Board

Re: Ask HN: Lessons learned from implementing user-facing analytics / dashboards?

#61

Post by bdcravens » Thu, Sep 28, 2023, 2:46 PM UTC

Users often catch what they see as conflicts, and you need to answer for this.

Often it's something as a different interpretation of data in multiple places (revenue in one place, profit in another) or differing date logic (one query includes a date in the range, others are "up to" that date, etc). Caching is another issue, especially if you selectively cache only slow queries.

To minimize this, always have an explanation on the chart/card (even if it's hidden but clickable to show)

Re: Ask HN: Lessons learned from implementing user-facing analytics / dashboards?

#62

Post by otronjones » Thu, Sep 28, 2023, 2:50 PM UTC

A common saying in statistical consulting is that the entire job is just asking "what question are you trying to answer?" over and over again.

Building dashboards that will actually be useful requires the same approach.

Re: Ask HN: Lessons learned from implementing user-facing analytics / dashboards?

#63

Post by BillFranklin » Thu, Sep 28, 2023, 2:52 PM UTC

As someone who has made a ton of grafana dashboards over the years, be prepared for users to hold it wrong. Data visualisations should fail/degrade in clear and expected ways. Users are often surprised when dashboards/charts hit some limit (eg they write a non performant query). The big query design (async first, fair queueing) is best if you’re letting users write their own queries on their own datasets.

Re: Ask HN: Lessons learned from implementing user-facing analytics / dashboards?

#64

Post by BillFranklin » Thu, Sep 28, 2023, 2:54 PM UTC

BillFranklin wrote: post #63

As someone who has made a ton of grafana dashboards over the years, be prepared for users to hold it wrong. Data visualisations should fail/degrade in clear and expected ways. Users are often surprised when dashboards/charts hit some limit (eg they write a non performant query). The big query design (async first, fair queueing) is best if you’re letting users write their own queries on their own datasets.

When I used to work with D3 I found object constancy to be quite an important principle. Transitions between state are often neglected (a full state refresh is easier).

Re: Ask HN: Lessons learned from implementing user-facing analytics / dashboards?

#65

Post by z5h » Thu, Sep 28, 2023, 3:11 PM UTC

Worked at a place providing financial research data and models to investors. We spent a lot of effort creating infinitely flexible and customizable reporting and dashboards. Turns out no one used that. Everyone just wanted a general high level report emailed to them.

Re: Ask HN: Lessons learned from implementing user-facing analytics / dashboards?

#66

Post by rc_mob » Thu, Sep 28, 2023, 3:12 PM UTC

z5h wrote: post #65

Worked at a place providing financial research data and models to investors. We spent a lot of effort creating infinitely flexible and customizable reporting and dashboards. Turns out no one used that. Everyone just wanted a general high level report emailed to them.

This sounds accurate. But of course the dashboards would be one of the biggest selling points in a sales pitch.

Re: Ask HN: Lessons learned from implementing user-facing analytics / dashboards?

#67

Post by kevan » Thu, Sep 28, 2023, 3:52 PM UTC

ravdeepchawla wrote: post #14

Biggest lesson: all metrics _must_ be defined in code, not manager-speak. For instance, if a marketing head wants to plot CAC (cost of acquiring customers) over time, saying CAC is number of customers divided by marketing spend is manager-speak. Spends are budgeted higher early in the month and adjusted with actuals. Customers ask for refunds and cancel accounts. Some campaigns have volume incentives which are known…

And make sure your project plan includes milestones for explicitly aligning all stakeholders on definitions otherwise you'll have a big hot potato game the first time the outputs show something unfavorable.

Re: Ask HN: Lessons learned from implementing user-facing analytics / dashboards?

#68

Post by tlarkworthy » Thu, Sep 28, 2023, 4:01 PM UTC

The phenomenal cost of hosting low latency realtime dashboards for everyone is a real cost. Tons of memory required if you want them to open quick for everyone. I wish they could be served more dynamically like if you saw a user loging, you could probably populate the query before they got to the page or something. As it was it seems like we have to serve a zillion dashboards noone is actively reading.

Re: Ask HN: Lessons learned from implementing user-facing analytics / dashboards?

#69

Post by gpm » Thu, Sep 28, 2023, 4:37 PM UTC

alexpetralia wrote: post #60

* Design matters a lot - if it looks bad, people won't look at it. * Layout for dashboards is almost completely formulaic. A panel for selected high-level stats (user growth % increase from last year, user % increase from last month, # new users added), a panel for breakdowns (user growth by marketing channel, user growth by registration cohort), a panel at the top for filters ("let's filter the entire dashboard by j…

> * Layout for dashboards is almost completely formulaic. A panel for selected high-level stats (user growth % increase from last year, user % increase from last month, # new users added), a panel for breakdowns (user growth by marketing channel, user growth by registration cohort), a panel at the top for filters ("let's filter the entire dashboard by just this marketing channel, or just this registration cohort") identical to all breakdowns provided, and finally a row-level drill-down ("show me the users in this particular aggregation"). It took me a very long time to learn that this design is entirely cookie-cutter for good reason. Users always want the same things: stats, breakdowns, filters and drill-downs.

Is there any chance you could link an image of what a good version of this looks like?

Re: Ask HN: Lessons learned from implementing user-facing analytics / dashboards?

#70

Post by jp_monteiro » Thu, Sep 28, 2023, 4:45 PM UTC

itsoktocry wrote: post #29

I've worked in many analytics projects across a number of companies as a consultant. I'm a big believer in "decision support systems". Find out what decisions your customers need to make, repeatedly, to their job. Quantify the heuristics and visualize that information (and that information only) in an easy to consume manner. More often than not that's an email or PDF . Another advantage is that by supporting the busi…

I agree that the term "self-service analytics" (especially the 'analytics' part) and "insights" just passes the wrong image of the real need of business users out there. It mixes 'strategic insights' with 'operational needs'. And I think self-service needs to be about operationalizing data. Sales managers are not necessarily looking to 'analyze' data or 'get an insight'. They need answers from data to manage their team. They need to track well-defined KPIs. See how their salespeople are doing and be able to have a productive meeting to tell them what they are neglecting. Customer success people need to "pull some data real quick" on the usage of the product by a certain client before a meeting.

These things happen all the time. And yet most companies out there think that the solution is to just build a bunch of dashboards, foreseeing what everyone will ask in the future. And then nobody checks the dashboards. Or finds the right one. And then they have a team of SQL translators pulling data for ad-hoc questions. That's silly IMO.

I'm obviously biased as a founder of a self-service analytics company based on AI (https://www.veezoo.com). But this is just my 2 cents on a topic I really care about.