Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Notes on the Perfidy of Dashboards

charity.wtf

Post reply on HN
Print view View original

11–20 of 62 posts

Re: Notes on the Perfidy of Dashboards

#11

Post by tetha »

SkipperCat wrote: post #8

Dashboards are invaluable. Humans can intake a lot of data from images and there is not better way to grok data than a graph. We've spent a lot of time building Grafana dashboards and they've been extremely helpful with debugging. It doesn't solve all problems but it certainly helps narrow down where to look. Sure, we still look at log files, use htop and a lot of other tools, but our first stop is always Grafana. I…

> We've spent a lot of time building Grafana dashboards and they've been extremely helpful with debugging. It doesn't solve all problems but it certainly helps narrow down where to look.

This is what I was about to write. Most of our services have 1 or 2 dashboards showing some service KPIs - for example HTTP request throughputh and response time, and also interface metrics to other sub systems - queries to postgres, messages to the message bus and so on.

With dashboards like this, you can very quickly build a deduction chain of "The customer opened a ticket, well because our 75%ile of the response time went to 20 seconds, well, because our database response times spiked to an ungodly number".

And then you can go to the dashboards about the database, and quickly narrow it down to the subsystem of the database - is something consuming the entire CPU of the database, is the IO blocked, is the network there slow.

In the happy cases, you can go from "The cluster is down" to "our database is blocked by a query" within a minute by looking at a few boards. That's very, very powerful and valuable.

And sure, at that point, the dashboards aren't useful anymore. But a map doesn't lose value because you can now see your target.

Re: Notes on the Perfidy of Dashboards

#12

Post by reureu »

I spent the past four years working as a data scientist for a healthcare company on population health initiatives, and started building out a body of research around how to engage clinicians using data (among other things, through dashboards). That's a bit different than the article, but one of my key learnings was that dashboards are often incredibly ineffective and only promulgated by well-intentioned engineers, based on what they would want to see if they were a clinician.

I worked with a behavioral economist, and we started running RCTs looking at different approaches to sharing data, and found that dashboards led to less engagement, when there was engagement it was more likely to drive ineffective interventions, and generally our dashboard groups had worse patient outcomes. Non-dashboard approaches had 20x better engagement and anywhere from 20-100% better patient outcomes (depending on the condition).

Unfortunately, both of us left the company when a bunch of engineers moved in, scoffed at our work, and immediately said "doctors need to be able to slice and dice their data" -- which, by every measure we had tested, is simply not true. But the "mission control" style thinking, where you have tons of dials and numbers flashing on a screen, pervaded because it "feels" like the right answer despite being the objectively wrong one.

Re: Notes on the Perfidy of Dashboards

#13

Post by spullara »

clipradiowallet wrote: post #2

From TFA... >every dashboard is a sunk cost >every dashboard is an answer to some long-forgotten question >every dashboard is an invitation to pattern-match the past >instead of interrogate the present >every dashboard gives the illusion of correlation >every dashboard dampens your thinking I disagree with this on all counts. A dashboard is a way to view multiple disparate metrics in a single place. Whether they are…

I mean, she is selling something. She is the founder (former CEO) of Honeycomb which is all about doing ad hoc queries rather than setting up dashboards.

Re: Notes on the Perfidy of Dashboards

#14

Post by cratermoon »

clipradiowallet wrote: post #2

From TFA... >every dashboard is a sunk cost >every dashboard is an answer to some long-forgotten question >every dashboard is an invitation to pattern-match the past >instead of interrogate the present >every dashboard gives the illusion of correlation >every dashboard dampens your thinking I disagree with this on all counts. A dashboard is a way to view multiple disparate metrics in a single place. Whether they are…

> A dashboard is a way to view multiple disparate metrics in a single place.

This is technically correct but doesn't approach anywhere near the criticisms the article has.

The deeper questions are: how did those metrics come to be collected, and why? What happened that resulted in those particular metrics being aggregated and displayed they way they are? What questions were being asked at the time the dashboards were created?

> a way to view multiple disparate metrics

So what? Why view them? Pretty graphs? A red/yellow/green? But to what end? This is why the statement is technically correct, but sheds no light at all on the reasons why a developer or customer support troubleshooter would care to look at the disparate metrics gathered in a dashboard.

Dashboards are created in response to certain problems and events. Those problems and events may or may not be relevant some time down the road. What happens when someone in the present with a certain set of questions or problems looks at the dashboard full of metrics capturing past questions and forgets that those questions are not today's questions?

Re: Notes on the Perfidy of Dashboards

#15

Post by drewcoo »

SkipperCat wrote: post #8

Dashboards are invaluable. Humans can intake a lot of data from images and there is not better way to grok data than a graph. We've spent a lot of time building Grafana dashboards and they've been extremely helpful with debugging. It doesn't solve all problems but it certainly helps narrow down where to look. Sure, we still look at log files, use htop and a lot of other tools, but our first stop is always Grafana. I…

A dashboard in a car gives you largely active feedback to help you drive. Mechanics' diagnostic tools give them a deep dive into information about repairing the vehicle, not limited to real time or to just driving activity. Those two things are very different.

I don't think a debugging "dashboard" is a dashboard. That conflates two very different ideas. We need a different name for that.

Re: Notes on the Perfidy of Dashboards

#16

Post by cratermoon »

SkipperCat wrote: post #8

Dashboards are invaluable. Humans can intake a lot of data from images and there is not better way to grok data than a graph. We've spent a lot of time building Grafana dashboards and they've been extremely helpful with debugging. It doesn't solve all problems but it certainly helps narrow down where to look. Sure, we still look at log files, use htop and a lot of other tools, but our first stop is always Grafana. I…

> We've spent a lot of time building Grafana dashboards and they've been extremely helpful with debugging. It doesn't solve all problems but it certainly helps narrow down where to look.

And then once the bugs that led to the creation of that dashboard are fixed or retired, what's left for that data? It just sits there with its pretty graphs and eye-catching visualizations to snare the unwary who are looking for help debugging a different problem. In fact, they'd be best served by ignoring existing dashboards and creating new ones specific to the issue in the present, not some dead husk of a problem that looks like it might be related.

Re: Notes on the Perfidy of Dashboards

#18

Post by lazyasciiart »

reureu wrote: post #12

I spent the past four years working as a data scientist for a healthcare company on population health initiatives, and started building out a body of research around how to engage clinicians using data (among other things, through dashboards). That's a bit different than the article, but one of my key learnings was that dashboards are often incredibly ineffective and only promulgated by well-intentioned engineers, ba…

Sounds interesting, is any of that body of research available publicly?

Re: Notes on the Perfidy of Dashboards

#19

Post by lazyasciiart »

cratermoon wrote: post #16
SkipperCat wrote: post #8

Dashboards are invaluable. Humans can intake a lot of data from images and there is not better way to grok data than a graph. We've spent a lot of time building Grafana dashboards and they've been extremely helpful with debugging. It doesn't solve all problems but it certainly helps narrow down where to look. Sure, we still look at log files, use htop and a lot of other tools, but our first stop is always Grafana. I…

> We've spent a lot of time building Grafana dashboards and they've been extremely helpful with debugging. It doesn't solve all problems but it certainly helps narrow down where to look. And then once the bugs that led to the creation of that dashboard are fixed or retired, what's left for that data? It just sits there with its pretty graphs and eye-catching visualizations to snare the unwary who are looking for help…

This is like advocating to get rid of log statements because they might not log the cause of the next bug.
Post reply on HN