Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In Praise of Observational Evidence

asteriskmag.com

Post reply on HN
Print view View original

31–40 of 56 posts

Re: In Praise of Observational Evidence

#31

Post by A_D_E_P_T »

estearum wrote: post #30
A_D_E_P_T wrote: post #12

Earlier quoted context omitted.

> "This feels off. In medicine, any evidence can also be blinded by confounding factors that are far easier to miss without adding specific controls. Really, in any field this will be the case." If you have enough data, you can smooth out individual fluctuations due to things like drug interactions, non-compliance, etc. (And indeed you might discover drug interactions!) Observational trials ultimately mirror how drug…

> Demands for increasingly byzantine trials This is silly. FDA has essentially one requirement: prove that your drug is safe and effective. The reason trial designs get more and more byzantine is because the drugs themselves work less-and-less well . They're far more nuanced and precise. The experiments have to be extremely well-controlled, and then this has to balance against cost/timeline of the trial, and that's w…

Proving "efficacy" -- which is the difficult and expensive part -- should not be necessary, and the government increasingly moves its own goalposts as to what the word efficacy even means. Simple as that. Postmarketing surveillance can easily determine what's effective and what's not, and medical orgs can adjust.

Re: In Praise of Observational Evidence

#32

Post by stronglikedan »

esafak wrote: post #25
KempyKolibri wrote: post #24

Earlier quoted context omitted.

Where is it defined as "Random Controlled Trial"?

Ignoring the typo above, it's mentioned in the text: https://en.wikipedia.org/wiki/Randomized_controlled_trial

Mentioned, yes, but not correctly. They should have put a "(RCT)" immediately after the first instance of "randomized control trial" at the very least. However, I'm of the opinion that the acronym should have been expanded it the first paragraph, or even first sentence, since it's used in the subheader. Either way, this was definitely a failure on the author's part.

Re: In Praise of Observational Evidence

#33

Post by graemep »

A_D_E_P_T wrote: post #12
taeric wrote: post #10

Earlier quoted context omitted.

This feels off. In medicine, any evidence can also be blinded by confounding factors that are far easier to miss without adding specific controls. Really, in any field this will be the case. Should we demand an RCT before we accept evidence? Of course not. At some point you do have to make a choice on things. And it should be noted that most drugs do have early cutoff criteria if the evidence is strong enough that it…

> "This feels off. In medicine, any evidence can also be blinded by confounding factors that are far easier to miss without adding specific controls. Really, in any field this will be the case." If you have enough data, you can smooth out individual fluctuations due to things like drug interactions, non-compliance, etc. (And indeed you might discover drug interactions!) Observational trials ultimately mirror how drug…

> Demands for increasingly byzantine trials have ballooned the costs associated with drug development,

Even if that is true, is it an intrinsic problem with trials or just bad regulation? If it is the latter then you need to change the regulations? Is the problem global - is every regulator everywhere demanding byzantine trails?

Re: In Praise of Observational Evidence

#34

Post by A_D_E_P_T »

timr wrote: post #21
A_D_E_P_T wrote: post #6

In medicine, observational evidence is actually better and far more ethical than the RCT. (Which simply dooms the terminally ill to fake treatment.) You just need large datasets and an agile culture that's responsive to new input. Don't forget that RCTs are very far from perfect and issues -- sometimes literally fatal issues -- have later turned up via observational evidence in large cohorts. Vioxx, for instance. Man…

> In medicine, observational evidence is actually better and far more ethical than the RCT. (Which simply dooms the terminally ill to fake treatment.) This is just nonsense. First, everyone in a trial is informed of the situation. It's not "unethical" unless you lie about it. If you participate in a trial, you do so knowing that you might not get the experimental drug. It's a selfless, honorable thing to do, and we s…

[flagged]

Re: In Praise of Observational Evidence

#35

Post by estearum »

A_D_E_P_T wrote: post #31
estearum wrote: post #30

Earlier quoted context omitted.

> Demands for increasingly byzantine trials This is silly. FDA has essentially one requirement: prove that your drug is safe and effective. The reason trial designs get more and more byzantine is because the drugs themselves work less-and-less well . They're far more nuanced and precise. The experiments have to be extremely well-controlled, and then this has to balance against cost/timeline of the trial, and that's w…

Proving "efficacy" -- which is the difficult and expensive part -- should not be necessary, and the government increasingly moves its own goalposts as to what the word efficacy even means. Simple as that. Postmarketing surveillance can easily determine what's effective and what's not, and medical orgs can adjust.

Brilliant idea. This way both actual working drugs and vitamin-aisle potions look identical to consumers. Neither (or both?) can make claims to their effectiveness since they're both backed by the same (lack of) actual knowledge.

This is especially great because it puts anyone who actually wants to actually make a working drug at a significant disadvantage. It'll take them longer to get to market, cost them a billion dollars more, then their medicine gets to sit next to a thousand variations of "Vitamin C for Leukemia" that all cost a lot less.

There would be virtually no incentive for anyone to make an actual drug.

> Postmarketing surveillance can easily determine what's effective and what's not, and medical orgs can adjust.

"Easily" is doing a ton of work. Postmarketing surveillance can sometimes give low-confidence signal as to what's effective and what's not.

Re: In Praise of Observational Evidence

#36

Post by estearum »

graemep wrote: post #33
A_D_E_P_T wrote: post #12

Earlier quoted context omitted.

> "This feels off. In medicine, any evidence can also be blinded by confounding factors that are far easier to miss without adding specific controls. Really, in any field this will be the case." If you have enough data, you can smooth out individual fluctuations due to things like drug interactions, non-compliance, etc. (And indeed you might discover drug interactions!) Observational trials ultimately mirror how drug…

> Demands for increasingly byzantine trials have ballooned the costs associated with drug development, Even if that is true, is it an intrinsic problem with trials or just bad regulation? If it is the latter then you need to change the regulations? Is the problem global - is every regulator everywhere demanding byzantine trails?

Regulators don't demand byzantine trials. They'd prefer simpler ones for a million different reasons. Byzantine trial designs exist because it's incredibly hard to prove your drug works even in an RCT. But GP thinks you can just look at EMR data and ta-da now you know, lol.

Re: In Praise of Observational Evidence

#37

Post by compiler-guy »

One classic paper for on this topic:

_Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials_

https://pmc.ncbi.nlm.nih.gov/articles/PMC300808/

"Advocates of evidence based medicine have criticised the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence based medicine organised and participated in a double blind, randomised, placebo controlled, crossover trial of the parachute."

Re: In Praise of Observational Evidence

#38

Post by mchusma »

Good article. I will only add that I think the problems outlined in the article are exacerbated by the regulatory apparatus. If this were a debate related to truth seeking alone, it would be good. But trials drive banning/unbanning of treatments and medicines, as well as their mandated coverage by insurance.

We could be much more flexible in our approach to things if we would un-ban things. For example after phase 1 trial or based on sufficient observational evidence, things can no longer be banned but have a higher standard for "insurance is now required to cover it".

Re: In Praise of Observational Evidence

#39

Post by CGMthrowaway »

derbOac wrote: post #11

Experimental designs are critical for obvious reasons but they have a few critical flaws, that mostly all reduce to the fact you can't randomize manipulations with everything. Whether it be due to ethics or practical constraints, you can't conduct a RCT all the time. This can be more subtly critical than it might seem, in that even if you can manipulate some proxy, often that proxy is insufficient in actually represe…

Yes. Even conceding every point made in this article, RCTs are still the gold standard. The article points out that RCTs are often impractical, expensive or ethically challenging. Well, so is the gold standard of anything.

OP suggests that alternative methods like target trial emulation, propensity scoring and double machine learning can be used to approximate the conditions of an RCT using existing data. In saying so he gives away the tell, which is that RCTs are the standard being aspired to.

Observational data may well be undervalued. But RCTs are still the gold standard.

Re: In Praise of Observational Evidence

#40

Post by qsera »

estearum wrote: post #35
A_D_E_P_T wrote: post #31

Earlier quoted context omitted.

Proving "efficacy" -- which is the difficult and expensive part -- should not be necessary, and the government increasingly moves its own goalposts as to what the word efficacy even means. Simple as that. Postmarketing surveillance can easily determine what's effective and what's not, and medical orgs can adjust.

Brilliant idea. This way both actual working drugs and vitamin-aisle potions look identical to consumers. Neither (or both?) can make claims to their effectiveness since they're both backed by the same (lack of) actual knowledge. This is especially great because it puts anyone who actually wants to actually make a working drug at a significant disadvantage. It'll take them longer to get to market, cost them a billion…

>There would be virtually no incentive for anyone to make an actual drug.

Dumb question: Wouldn't the market discover what works and what does not automatically, like any other product?

Post reply on HN