Live data from Hacker News

Notes on the Perfidy of Dashboards

charity.wtf

31–40 of 62 posts

Re: Notes on the Perfidy of Dashboards

#31

From TFA... >every dashboard is a sunk cost >every dashboard is an answer to some long-forgotten question >every dashboard is an invitation to pattern-match the past >instead of interrogate the present >every dashboard gives the illusion of correlation >every dashboard dampens your thinking I disagree with this on all counts. A dashboard is a way to view multiple disparate metrics in a single place. Whether they are…

> A dashboard is a way to view multiple disparate metrics in a single place. This is technically correct but doesn't approach anywhere near the criticisms the article has. The deeper questions are: how did those metrics come to be collected, and why? What happened that resulted in those particular metrics being aggregated and displayed they way they are? What questions were being asked at the time the dashboards were…

That's why good dashboards come with Title, subtitle, legend, the X and Y axis, and units. Count of packets denied from source IP, source port, last 4 hours. Average number of requests forwarded to proxy farm, distributed by server, last 7 days vs. same time last month. Who called 2049, last 24 hours.

Re: Notes on the Perfidy of Dashboards

#32
post #12

I spent the past four years working as a data scientist for a healthcare company on population health initiatives, and started building out a body of research around how to engage clinicians using data (among other things, through dashboards). That's a bit different than the article, but one of my key learnings was that dashboards are often incredibly ineffective and only promulgated by well-intentioned engineers, ba…

Are you me? I also built out a body of research in a healthcare company about how to engage clinicians using data, that stopped us from making some mistakes until we got acquired. No doctor (or chief of staff, or chief medical officer, or anyone besides an actual data analyst) will slice and dice their data. Doctors won't even listen to a non-peer clinician discuss data with them, let alone their administrators.

In my timeline, I told the engineering team that all of their work was almost certainly for naught, that all the research said this product would completely fail, and we were basically just doing it for managerial and contractual obligations.

This gave engineering the freedom to use whatever technologies and conduct whatever technical experiments they wanted, since no-one would ever use the product, and it'd likely be shut down soon after launch for disuse.

A key hospital partner gave us a couple dozen docs to test it with. I interviewed them about how they measured their work and impact, and the data they used to improve their craft and outcomes. I asked them to review every measure on the dashboard, explain their understanding of it, and explain how their work or behavior would change based on that.

Almost to a person, the doctors said there was nothing of use to them there, as the research predicted. Some of these doctors were on the committee that specified the measures they would themselves be seeing.

The product was launched with great managerial acclaim, and promptly sunset 12 months later from disuse.

Re: Notes on the Perfidy of Dashboards

#33
post #17

I read this and still don't know exactly what the author is asking for. Is an "exploratory, queryable interface" not exactly how you would describe a modern dashboard?

They want you to use their interactive dashboards backed by their database using features their PMs prioritized, like wide columns and drill-downs. But you're right, other systems do support these . Ex: Splunk has done wide columns with index-on-write columnar analytics and point-and-click pivoting/drilldowns since ~day 1. So, as they add features, they get very definitional on each one. Most startup people (myself included!) have a lot to learn from their successful developer/IT marketing style.

Re: Notes on the Perfidy of Dashboards

#34
post #21
post #12

I spent the past four years working as a data scientist for a healthcare company on population health initiatives, and started building out a body of research around how to engage clinicians using data (among other things, through dashboards). That's a bit different than the article, but one of my key learnings was that dashboards are often incredibly ineffective and only promulgated by well-intentioned engineers, ba…

Curious if you have any examples of "Non-dashboard approaches" to compare and contrast?

We tried a couple of different approaches: tableau reports, emailing static data to providers, sending spreadsheets of patient-level data, and building a Facebook or Twitter style feed. And then had different variations on each, and would run trials comparing different approaches.

We pretty quickly found that sending data ("push") was way more effective at engagement than just having a tableau report they could go to ("pull"), even when that dashboard was linked directly within the EHR, didn't require a login, and was contextualized for the provider (basically as low friction as you could get-- they would actually venture into it 1-2 times per year).

We ran a trial where we changed how we presented data: either in terms of number of patients ("screen 20 people for depression this week") or in terms of quality rates ("your depression screening rate is 40% and going up"). Keeping the data in terms of patients led to ~20% improved screening, and in the surveys led to providers expressing more trust in the data (although, they also were more likely to say they "didn't have enough data to do [their] job", despite actually doing their job better than the other group).

So then we took that idea for depression screening and extended it from depression screening to chronic disease management (where the specific task for the provider is much more variable). So we had one arm where we gave them access to data marts and trained them on how to "slice and dice" the data, and then compared that against a newsfeed that had the data pre-"sliced and diced". The engagement was higher in the newsfeed group. Interestingly, the only thing the "slice and dice" group seemed to do was look for patients without a primary care doc designed in the EHR and just assign them-- in evaluating the outcomes for this, that was the single least effective intervention they could do to improve chronic disease care (and this was validated in a follow-up study looking explicitly at the impact of PCP-assignment on patient care). So, our "newsfeed" arm ended up with, on average, around 60% better outcomes than the "slice and dice" arm.

What's funny is that through all of this, some of the leaders would also say "we need more data!!" But when we'd build a tableau report for them, they'd use it once or twice and then never again. Or, in one case, the leader would actually use it for ineffective purposes ("we have to assign them all PCPs!!") or for things that are easily automated ("we're going to email each of our uncontrolled hypertensive patients"). I firmly believe that for doctors and data, you need to have clearly defined objectives: the goal should never be "give them access to data", but rather should be something like "make providers feel like they have the data necessary to do their job" and "improve quality rates through targeted provider intervention." Start from those first principles, and validate your assumptions at each step. I'm confident your solution won't end up with tableau.

Re: Notes on the Perfidy of Dashboards

#35
post #12

I spent the past four years working as a data scientist for a healthcare company on population health initiatives, and started building out a body of research around how to engage clinicians using data (among other things, through dashboards). That's a bit different than the article, but one of my key learnings was that dashboards are often incredibly ineffective and only promulgated by well-intentioned engineers, ba…

Are you me? I also built out a body of research in a healthcare company about how to engage clinicians using data, that stopped us from making some mistakes until we got acquired. No doctor (or chief of staff, or chief medical officer, or anyone besides an actual data analyst) will slice and dice their data. Doctors won't even listen to a non-peer clinician discuss data with them, let alone their administrators. In m…

I dunno, but it sounds like we should get a beer sometime.

Not sure if this resonates also, but the engineers that took over all came from outside healthcare and had a strong "I'm going to apply what I know from Ticketmaster to solve healthcare!" mentality. Those of us that have 15 years of experience in healthcare would, at best, have a 10 minute "knowledge sharing" meeting with the engineering and product managers. And then we'd sit back and watch them make some really naive mistakes. [to be clear, I'm not about gatekeeping people from being involved in health tech, but rather I'm just exhausted at interacting with people with no self-awareness about the amount of things they don't know about a particular domain]

I'm still a bit bummed because I think we were actually just starting to get to some really cool, actually innovative, population health approaches that seemed effective for both improving outcomes and minimizing provider burnout. :(

Re: Notes on the Perfidy of Dashboards

#36
> That’s not debugging, that’s pattern-matching. That’s … eyeball racing.

Um...yes. And that is a very good thing. Because if there is anything the human brain is good at, it's pattern matching. Especially on visual data.

It's an extremely quick and efficient way to find out where to start the detailed debugging.

And there is a lot of value in that.

Re: Notes on the Perfidy of Dashboards

#37
post #12

I spent the past four years working as a data scientist for a healthcare company on population health initiatives, and started building out a body of research around how to engage clinicians using data (among other things, through dashboards). That's a bit different than the article, but one of my key learnings was that dashboards are often incredibly ineffective and only promulgated by well-intentioned engineers, ba…

[deleted]

Re: Notes on the Perfidy of Dashboards

#38
post #34
post #21

Earlier quoted context omitted.

Curious if you have any examples of "Non-dashboard approaches" to compare and contrast?

We tried a couple of different approaches: tableau reports, emailing static data to providers, sending spreadsheets of patient-level data, and building a Facebook or Twitter style feed. And then had different variations on each, and would run trials comparing different approaches. We pretty quickly found that sending data ("push") was way more effective at engagement than just having a tableau report they could go to…

Thanks for writing this up, it's a valuable resource that can be applied to other industries.

My takeaways:

- Start with first principles, such as "improve quality rates through targeted provider intervention"

- Push and simple stats works better vs. pull with fancy dashboards

- Slice and dice can help identify process exception but not great for process improvement, whereas simple stats on a regular basis improve outcomes

Re: Notes on the Perfidy of Dashboards

#39
post #12

I spent the past four years working as a data scientist for a healthcare company on population health initiatives, and started building out a body of research around how to engage clinicians using data (among other things, through dashboards). That's a bit different than the article, but one of my key learnings was that dashboards are often incredibly ineffective and only promulgated by well-intentioned engineers, ba…

Sounds interesting, is any of that body of research available publicly?

Some, not all. I'm hesitant to directly post links on here since it will out both me and the company I worked for. If you're interested in this kind of work, you should check out the conference proceedings for AMIA (amia.org)

Re: Notes on the Perfidy of Dashboards

#40

From TFA... >every dashboard is a sunk cost >every dashboard is an answer to some long-forgotten question >every dashboard is an invitation to pattern-match the past >instead of interrogate the present >every dashboard gives the illusion of correlation >every dashboard dampens your thinking I disagree with this on all counts. A dashboard is a way to view multiple disparate metrics in a single place. Whether they are…

I mean, she is selling something. She is the founder (former CEO) of Honeycomb which is all about doing ad hoc queries rather than setting up dashboards.

Get 'em
Post reply on HN