Notes on the Perfidy of Dashboards

Not the OP, but in my experience, when clinicians ask for more data, they're actually asking for more lineage or provenance metadata.

We boiled it down for our teams like this:

Administrators are top-down: they want a high-level view and to be able to drill down from there.

Individual physicians are bottom-up: they want "their" data about "their" patients, and maybe, sometimes, to compare to the peers they personally trust.

As with any professional group, there's some minority percentage that treats their work like a craft and knows how to use data to improve their practices; but the majority want qualitative data and value interpersonal relationships. Giving a dashboard to the latter group at all is wasting time and effort of all parties.

If your dashboard can't attribute all of its data and all of the patients referenced to match the physician's definition of "theirs," you've lost. That's the "more data" and "drill down" physicians care about.

If your dashboard isn't timely and clinical -- which generally means presented in a clinical voice, at the point of care, or otherwise when they have an opportunity to make the change you want them to make -- it's not going to be actionable. That means surfacing some alternative action right before they see a patient which might benefit from that, which is not when they're on their computer. They might be one of those doctors that never is on their computer until the very end of the day. Looking at your dashboard at 11pm about the patients from earlier today (or more likely, earlier this past quarter of the year) is not helpful.

Looking at your dashboard is non-clinical work, and doctors want to do clinical work. If you're going to make them go do a new and non-clinical thing, it has to reduce some other non-clinical thing in a way that's meaningful to them. Otherwise, they're just as likely to do an end-run around your application entirely, like the doctors who only use index cards to write notes or who fail to enter their passwords every morning and lock themselves out, so they don't have to use the EMR.

With respect to the tldr - so, on one arm, so to speak, is an interface where clinicians could use different combinations of measures to identify patients who might need some kind of intervention, like follow up attention. Then the other side is an analytic system that uses a specific set of measures, etc, and then messages clinicians with a recommended set of actions?

In the first case, the clinicians have to do analytical work (slice, dice) towards understanding the population of patients. That sounds more like epidemiology... In the latter case, how is it that clinicians will trust the recommender? Is it understood that there is a clinical rationale or authority behind the algorithm? It sounds like "uncontrolled" in this case is based on a measure that clinicians trust.

I think of dashboards as potentially good for monitoring outcomes against expectations, EDA as potentially good for focusing attention on subpopulations, and recommenders as potentially good for efficiently allocating action. In a broad way what you described is a monitoring system that pushes recommended actions out to doers. I'd venture that with busy clinicians that that needs to be pretty accurate, too, and/or that recommendations need both explicit justification and a link to collateral information.

Honestly, the complaint here seems to be less about dashboards and more about the data behind them.

Static dashboards sound like timeseries backends, where the data is pre-aggregated (graphite / statsd, prometheus). You can't really drill down into the metrics, or can only drill down into preplanned dimensions. Grafana is a commonly used dashboarding frontend here.

Dynamic dashboards, in contrast, are dynamically aggregating data. More akin to structured logging, or maybe splunk / ELK. You have granular data, and write queries to extract, filter, and aggregate on demand. Tableau, PowerBI, Apache Superset all compete in this space.

But by focusing on the dashboard angle, the reader doesn't think to hardly about why they're different, and also why you might prefer one over the other. TSDB like Prometheus are very fast, and if you focus on collecting aggregate data, allow you to collect a lot more metrics, or sample much faster. You're probably not logging in the TSDB any labels associated with UserAgent strings, or screen size your mobile app got, etc. By paying the price in dimensionality, you get much faster queries at lower cost. I'll let you guess which type of backend Charity's startup represents.

Both have a place. I've been able to build canary dashboards that work quite well using both backends, as a proof of concept that something like Kayenta is feasible for my team. In fact, high dimensionality works against you in release engineering. The more dimensions you can compare across, the higher chance for a false positive, and the more investigations engineers have to do to rule them out. Worse, there are often confounding variables you need to go hunting for, and the dashboard won't find them for you.

And execs absolutely don't want to have to care about the complex causality chain you need to model. They want 'a single number' to improve on over time. They don't want a dashboard to dive in and analyze on ten dimensions. They want to see their chosen KPIs going to the right and up. Fundamentally, the dashboard is less important than your audience.

Quality measures are generally well-defined by external authorities, so questions like "what defines uncontrolled" are generally answered. Even when providers personally disagree with this (I worked with a provider who didn't believe in pre-diabetes), they still acknowledge that health care organizations are being judged on these measures, and that the measures are not just arbitrarily defined. How you improve your quality measures becomes where the question turns.

Your comment about epidemiology/EDA/etc really hit the nail on the head. If you sit in on population health meetings at your average hospital/clinic system, you'll see that many people really don't get this. Further, people often conflate their needs/desires with that of others-- so, the data-driven administrator is quick to say "we just need doctors to be able to slice and dice their data, and then we'll have better quality scores." But they're talking about what their needs are, and it's completely not what the doctors actually need (well, and from monitoring usage of dashboards for those types, I'd argue it's also not what they need either, but that's a different issue). And, the reason I keep saying "slice and dice" is because I've heard that phrase used by every vendor I've evaluated, and in practically every strategy meeting regarding population health at multiple institutions.

I'd personally shy away from describing this issue in terms of a recommender, since that has a pretty connotation in the ML world, and it doesn't really line up well (e.g., there's not a well-defined objective function or a clear feedback loop to train a recommendation system on). However, getting away from that specific concept, I think it's reasonable to say that there are needs for multiple distinct but ideally-related systems in the population health world: one for analysis to be used by quality and data people, and one specifically for the clinicians doing the work.

No, it's like advocating to get rid of log statements because they are like an overflowing sewer of data obscuring and distracting from the real cause of the bug. But you can't even figure out if there's a message in the log because it takes an hour to sort through the firehose of messages you know aren't relevant, but there they are, filling up your logs.

The above goes double if you're running Java and your logs include stacktraces spanning 300 lines.