Nearly 400 medical devices, procedures and practices found ineffective in study

What is the best learning rate for updating physicians (our models) from the results of RCT's (part of our loss)?

The authors reviewed all articles in three journals from (generally) between 2003-2017. They didn't, afaict (please point me if they did), review the time-to-correction (if any correction has been made). It takes some time before the results of an RCT end up in established practice. I'm actually surprised it's so small in many cases.

It's not like there's a database where the results of every RCT are immediately updated and the physician model is retrained overnight on the new data.

Even if there were, imagine if the learning rate (so to speak) were so high that every discipline immediately changed their published best practices on the basis of a single RCT?

Here's a cautionary paragraph from one of the excellent reversal studies they use:

Several limitations of the study warrant discussion. First, because we enrolled only 26% of eligible patients, our findings must be generalized cautiously. The most frequent reason that patients declined enrollment was a strong preference for one treatment or the other. Since patients' preferences may be associated with treatment outcome, our trial may be vulnerable to selection bias. Participating surgeons may not have referred potentially eligible patients because they were uncomfortable randomly assigning these patients to treatment; this form of selective enrollment may also create bias.26 Second, because the trial was conducted in academic referral centers, the findings should be generalized carefully to community settings. Third, we did not formally assess the fidelity of the physical therapists or surgeons to the standard intervention protocols. Finally, our study was not blinded, since our investigative group did not consider a sham comparison group feasible. [0]

I'm less concerned about RCT to Best Practice time than from Best Practice to Typical Physician Practice time. There is a cascaded model connected to the 'complex RCT loss' and it's discipline published practice down to individual physician treating patients. Compressing the time from RCT to individual physician is fraught with difficulties, but could be improved.

Finally, RCT is the gold standard, but it's not perfect and it doesn't always clearly translate to the individual physician's model of practice. Many best practices weren't established from RCT's either.

And an inconclusive result from an RCT is not the same thing as proving that there's no difference in outcomes, but a proper statistician can chime in there.

[0]https://www.nejm.org/doi/full/10.1056/NEJMoa1301408

* Fixed a bug in an existing "legacy system", deployed remotely without your knowledge. You have no logs or debug information, but you can speculate on cause and try it in small simulated / test environments (e.g. rodents). By introducing this fix you would also be changing the system in unknown ways.

* You only have a vague idea of how the code in the OS for this system works, your fix doesn't even change the code. Your only mechanism of change is injecting some foreign JS into a webform to cause ripple effects on the system for the user, meanwhile the system is setup to try to stop / avoid such "attacks" and will try to undo your fix if it can be detected by the system's "AI".

* if your fix fails in production for even one user, you may lose millions and potentially go jail, that person may die.

* You get one shot to release it to the public, after go-live it basically has to work or everything was for nothing. There is no "push this fix and everything gets better", even one small change is a complete restart (new molecule == new drug).

* If you accidentally release any information about the users, or lose their data, you lose millions and potentially go to jail.

* This software will take 10 years, a handful of scientists, a lab, some lawyers, lobbying, and a few billion dollars to make.

* Every user using your software will be on an entirely unique stack, processing all kinds of data in tandem with your software, and will not follow directions. Some will even have lots of other bugs in the system that are unrelated to your fix.

The anti vaxxers lack of trust in the media has almost nothing to do with what the media may or may not have done to lose trust.

The actual difference is that one side does not have a good explanation for what causes autism, and as a result admits that they don’t have the answers yet, while the other does not have a good explanation as well, but makes up stuff and boasts that they actually do know the answer.

For vulnerable parents who are just looking for some answer, the second side is extremely attractive.

Everything else about lack of trust, and corporate influence, etc is all secondary, and are after the fact justifications.

In dentistry/endodontics it was believed for a long time that installing a post helped keep a crown and tooth intact for longer. But long term studies showed posts can actually reduce longevity. In another case, a study showed that a common type of knee surgery for pain is actually less helpful than a placebo surgery.

The overarching problem is that it's (currently) very hard to evaluate the real effect of practicing medicine on large swaths of the population, and then using those results to change what doctors do. By getting doctors into a continuous improvement cycle, we could identify costly, useless practices, and improve health.