Heart surgeons refuse difficult operations to avoid poor mortality ratings • Hacker News Board

Re: Heart surgeons refuse difficult operations to avoid poor mortality ratings

#91

Post by Buttons840 » Sat, Apr 21, 2018, 7:35 PM UTC

fc003153 wrote: post #73

cs702 wrote: post #30

It's Goodhart's Law in action, this time with deadly consequences: "When a measure becomes a target, it ceases to be a good measure."[a] [a] https://en.wikipedia.org/wiki/Goodhart%27s_law

I'm not so sure Goodhart's Law is correct. For example, if I target a high win percentage, is win percentage, then, not a good measure? Mortality rates are a pretty good measure for surgeons. The problem here is that they are inflating their measure by cherry picking. edit: Some measures should be targeted, the problem here is that it is done in an non genuine way

Pro tip for those targeting a high win percentage. If you win your first game (beginners luck?), then stop playing and never play again.

That's what the article is describing, surgeons won't even play the game, they won't even attempt to help certain people because it might hurt their "win percentage".

Re: Heart surgeons refuse difficult operations to avoid poor mortality ratings

#92

Post by credit_guy » Sat, Apr 21, 2018, 7:37 PM UTC

samfisher83 wrote: post #20

Sports athletes do this. For example players won't attempt end of the quarter half court shots since that will lower their percentages, or QBs taking sacks to not lower their QB ratings. What they do in sports is keep advanced stats. Instead of just giving a pure mortality rating have mortality rating, by surgery type.

Maybe in other sports this happens, but I seriously doubt QB's engage in that. First of all, taking a sack is not pleasant. In the extreme it can take you out of the game, or land you on the IR (injured reserve). Drew Bledsoe was replaced by Tom Brady after such a sack, and was never the star he had been before. Second, the number of sacks a QB takes is a statistic in itself. Third, the top stats for a QB are in this order: touchdowns, interceptions, passing yards, and completion percentage. If a QB takes a sack to avoid an interception, I'm 100% happy with that. If he takes a sack so that his completion percentage is higher by 0.01%, I would not be, but then the completion percentage is reported at the 0.1% level of precision. There is simply too much physical pain in taking a sack for a QB to trade that for an imperceptible gain in a pretty unimportant stat.

Re: Heart surgeons refuse difficult operations to avoid poor mortality ratings

#93

Post by WalterSear » Sat, Apr 21, 2018, 7:37 PM UTC

bradlys wrote: post #11

firstplacelast wrote: post #3

Check out the comments over on /r/medicine and you will likely be horrified. It is career over everything. There was even a thread this week about how you should NEVER correct someone above you (med students should never correct a resident and a resident should never correct an attending). That mentality is terrible for everyone and every thing, except the physicians' careers. I don't know if we need to select for be…

If you think about what it takes to get into med school then it seems pretty natural that you're going to get hyper competitive surgeons in the end. It starts before med school. You get assigned 10 unobtainable books that only the library has. Finals week comes around, all the books are missing even though they have to stay in the library. Turns out your classmates hid them inside the library so they could have a com…

Hyper competitive, conformist surgeons.

Re: Heart surgeons refuse difficult operations to avoid poor mortality ratings

#94

Post by flamedoge » Sat, Apr 21, 2018, 7:37 PM UTC

kazinator wrote: post #44

cs702 wrote: post #30

It's Goodhart's Law in action, this time with deadly consequences: "When a measure becomes a target, it ceases to be a good measure."[a] [a] https://en.wikipedia.org/wiki/Goodhart%27s_law

Low mortality is a good damned measure. I don't want to be operated on some yahoo surgeon who unconditionally operates regardless of risk and has a high mortality track record as a result.

So if it's difficult we shouldn't try?

Re: Heart surgeons refuse difficult operations to avoid poor mortality ratings

#95

Post by washadjeffmad » Sat, Apr 21, 2018, 7:38 PM UTC

fc003153 wrote: post #73

cs702 wrote: post #30

It's Goodhart's Law in action, this time with deadly consequences: "When a measure becomes a target, it ceases to be a good measure."[a] [a] https://en.wikipedia.org/wiki/Goodhart%27s_law

I'm not so sure Goodhart's Law is correct. For example, if I target a high win percentage, is win percentage, then, not a good measure? Mortality rates are a pretty good measure for surgeons. The problem here is that they are inflating their measure by cherry picking. edit: Some measures should be targeted, the problem here is that it is done in an non genuine way

Wins are likely from systems with fixed rules that don't reward innovation, so winning by the rules of a system doesn't necessarily mean a positive outcome. An example might be the system that lead to the housing crisis. Certainly a few people who gamed things the right way "won" big, but only at the expense of millions of homeowners.

Mortality rates are one measure of many that can tell a story, but they're like incarceration rates in that they've become a perverse incentive that exacerbates an issue. If we say that putting more people in prison is a good thing, that's predicated on an assumption that only those that are being put in prisons are people worthy of being there, whose freedom is more costly to society than their imprisonment. When a district is rewarded for putting people in prisons, what types of things might you expect to happen to the legal systems and populations in those places? Would prosecutors then be incentivized to trump up charges and force people into serving prison time unnecessarily to make themselves look like bigger "winners"? Might police make more frivolous arrests or even stir up trouble in communities to paint a picture of rampant criminality so they can look to be "tough on crime"? Wouldn't you expect to see an all-causes decrease in long term crime?

The problem isn't just cherry picking, but if people can game a system built around limited and gameable measures, then it's going to encourage min-maxing for profit.

Re: Heart surgeons refuse difficult operations to avoid poor mortality ratings

#96

Post by JulianMorrison » Sat, Apr 21, 2018, 7:38 PM UTC

cs702 wrote: post #30

It's Goodhart's Law in action, this time with deadly consequences: "When a measure becomes a target, it ceases to be a good measure."[a] [a] https://en.wikipedia.org/wiki/Goodhart%27s_law

That only really applies when you're measuring side effects though, and not the actual thing intended.

In this case they should have had a metric that took into account the estimated prior probability of a good outcome given an ordinary surgeon.

Re: Heart surgeons refuse difficult operations to avoid poor mortality ratings

#97

Post by LoSboccacc » Sat, Apr 21, 2018, 7:39 PM UTC

fc003153 wrote: post #73

cs702 wrote: post #30

It's Goodhart's Law in action, this time with deadly consequences: "When a measure becomes a target, it ceases to be a good measure."[a] [a] https://en.wikipedia.org/wiki/Goodhart%27s_law

I'm not so sure Goodhart's Law is correct. For example, if I target a high win percentage, is win percentage, then, not a good measure? Mortality rates are a pretty good measure for surgeons. The problem here is that they are inflating their measure by cherry picking. edit: Some measures should be targeted, the problem here is that it is done in an non genuine way

the problem here is that mortality rate is not correlated/normalized with the patient risk factor. the problem lies in the performance indicator itself, not strictly people trying to optimizing for what they are asked for. it is likely inhumane to refuse patients that have slightly above average risk - even if there are some considerations to be had, like having replacement organs go to waste, but that is already covered by the donor lists queues - but given people can find themselves out of work for having bad mortality rates, optimizing for the performance indicator it's the only logical conclusion.

also, normalizing for the patient risk will likely just lead to people overstating patient risk, because once you reduce performance to an index, people will focus on what's measured before on what the intended goal for the measurement is.

Re: Heart surgeons refuse difficult operations to avoid poor mortality ratings

#98

Post by setr » Sat, Apr 21, 2018, 7:40 PM UTC

kazinator wrote: post #44

cs702 wrote: post #30

It's Goodhart's Law in action, this time with deadly consequences: "When a measure becomes a target, it ceases to be a good measure."[a] [a] https://en.wikipedia.org/wiki/Goodhart%27s_law

Low mortality is a good damned measure. I don't want to be operated on some yahoo surgeon who unconditionally operates regardless of risk and has a high mortality track record as a result.

You presumably want the metric (operation, mortality rate); its useless to consider mortality across the board, because that assumes all activities are equally risky. Otherwise, you end up like this, just entirely removing the option of high-risk activities, regardless of the potential benefit.

Essentially, the doctor has decided that he doesn't want to take the risk of killing you, even if you're willing to accept it. Why should that be the doctor's choice to make, and why is he being given this additional risk, when the benefit/loss should clearly be yours?

Re: Heart surgeons refuse difficult operations to avoid poor mortality ratings

#99

Post by olliej » Sat, Apr 21, 2018, 7:40 PM UTC

cs702 wrote: post #30

It's Goodhart's Law in action, this time with deadly consequences: "When a measure becomes a target, it ceases to be a good measure."[a] [a] https://en.wikipedia.org/wiki/Goodhart%27s_law

It also kind of incentivizes the alternative as well: if you aren’t an amazing surgeon, you could work to ensure you exclusively work on “hard” cases you can claim all failures are due to patient difficulty rather than lack of competence (at least competence to deal with that difficulty). Then you can dismiss any claims you’re incompetent with “I have a high mortality rate because I’m willing to risk my reputation to save people who have been abandoned”...

Re: Heart surgeons refuse difficult operations to avoid poor mortality ratings

#100

Post by billfruit » Sat, Apr 21, 2018, 7:41 PM UTC

fricat1ve wrote: post #79

billfruit wrote: post #12

On a related vein, do surgical procedures involve any type of quality control, where the work is independently audited? If so I wasn't aware. Quality control is such an important cogwheel of processes in many other industries, yet I don't hear much talk of it in relation to medical profession.

In the US, if the surgery is in hospital: - Well-regarded hospitals will vet surgeons before granting privileges. - Average hospitals give out privileges fairly easily if there are no actions against a person's license. - There is a "collegial" review of big screw-ups that carry major reputational risk. - As with any fee-for-service firm, "rainmakers" are highly sought after and get away with more. - Privileges are d…

I was thinking something more akin to routine inspection program carried out by independent personnnel applicable to all procedures, not just the reviews that are necessitated due to screw ups.