How many medical studies are faked or flawed? • Hacker News Board

Re: How many medical studies are faked or flawed?

#221

Post by Llamamoe » Wed, Sep 20, 2023, 7:57 AM UTC

riperoni wrote: post #214

strken wrote: post #177

Earlier quoted context omitted.

"Trust the science" is dumb, but "trust that the single-most reviewed medical studies of the past decade with 30k+ participants aren't outright bullshit" is enough for me. The risk for those studies is probably closer to what happened with Astrazeneca, where a very rare side effect got missed because 30k people weren't enough to detect it. I think skepticism is healthy to a degree, but ironically, the more adversaria…

Severe side effects or death cases were excluded from the Pfizer phase 3 study by conveniently shifting blame on patients, inventing diagnoses or claiming they had a Covid infection. Even if all that were not intentional, it would certainly raise the question why this should be the only case of fabricating data for studies. Since the RNA-vaccines are obviously not as positively effective as claimed, what else is ther…

Early into the pandemic, the original COVID strains had like 40% rate of permanent aftereffects, and the hospitalisation and death rates were insane. At the time, the vaccines have definitely saved lives, and the "covered up severe side effects or death" you talk about never came up in the real world, so...

Re: How many medical studies are faked or flawed?

#222

Post by somenameforme » Wed, Sep 20, 2023, 8:22 AM UTC

fer wrote: post #220

somenameforme wrote: post #216

Earlier quoted context omitted.

Whether the original study said so or not, that is absolutely how it was universally sold to the public. This [1] is a collection of various high profile individuals talking about the efficacy of the vaccines, why that means you won't carry the virus, you won't get sick, and and how that will completely stop the spread of COVID. It's only as this very obviously failed to be the case that the metric was completely shi…

> Whether the original study said so or not, that is absolutely how it was universally sold to the public. Trust the politics, blame the science?

By "the politics" you're including individuals like Fauci, the head of the CDC, and so on. These are individuals that were perceived as "the science." Beyond this, a peer comment motivated me to look up the original trial results and reporting. And yeah, it was entirely about cases. Here [1] are the data from Moderna. Oddly enough, it seems to have been moved, if not removed, from their website, but fortunately nothing ever dies on the internet.

"There were 11 COVID‑19 cases in the Moderna COVID‑19 Vaccine group and 185 cases in the placebo group, with a vaccine efficacy of 94.1% (95% confidence interval of 89.3% to 96.8%)."

They were claiming it outright prevented COVID, as vaccines generally do. So "the politics" and "the science" were in lockstep on this one. As were they when they seamlessly dropped this narrative and swapped over to hospitalization/death.

[1] - https://web.archive.org/web/20210202223626/https://www.moder...

Re: How many medical studies are faked or flawed?

#223

Post by botanical » Wed, Sep 20, 2023, 8:35 AM UTC

Off-topic but how do you archive a page that is paywalled but you have personal access to it by logging in? How was this article archived?

Re: How many medical studies are faked or flawed?

#224

Post by ghomem » Wed, Sep 20, 2023, 8:36 AM UTC

This topic concerns me a lot, since the Ivermectine phenomenum plus the Covid Vaccines. People mentioned sample sizes in several comments. What concerns me is that sample size requirements versus incidence of a problem (ex: covid deaths) is a very non-intuitive discussion for the general public, and even sometimes for people in technical fields.

Here is a simple tool that can help exploring sample sizes:

https://lo.gic.li/lgc

Re: How many medical studies are faked or flawed?

#225

Post by mike_hearn » Wed, Sep 20, 2023, 9:17 AM UTC

whatshisface wrote: post #40

some_random wrote: post #35

Earlier quoted context omitted.

This goes far beyond there not being enough replication studies, the problem is that when replication studies are done the results don't reproduce because the results were bogus. https://en.wikipedia.org/wiki/Replication_crisis

If the replication studies were done at a reasonable rate there would be no incentive to produce bogus results because you'd be caught before you could go through an entire career as a "successful scientist."

There is no being caught :( Who is going to do the catching, exactly? There's no science police. Universities don't seem to care about anything except plagiarism.

Many papers aren't even "looks legit but doesn't replicate", they are self-evidently wrong just from reading them or the associated data. Just peer review done properly would solve that, but the problem is widespread because there's no real incentive to rock the boat. Once a bad technique or invalid approach gets through to publication a few times it becomes a new standard within the field because it lets people publish more papers.

Re: How many medical studies are faked or flawed?

#226

Post by mike_hearn » Wed, Sep 20, 2023, 9:20 AM UTC

LudwigNagasena wrote: post #89

RugnirViking wrote: post #46

Earlier quoted context omitted.

> the scientific community's reaction to the replication crisis has been to ignore it What can they do? It's an incredibly hard problem to solve. It's like asking why the buisness community has done nothing to adress the housing crisis. Large scale culture changes, or the entire structure of the way science is conducted and funded, would be the only solutions

It doesn’t have to be large scale. It can start with a single journal, with a single university, with a single scientist. Make peer review public. Weight replication studies more. Make conducted peer reviews and replications an important metric for tenure and grants. Publish data and code alongside papers.

Some of those things were tried already. Some peer reviews are public, I've read them. Data and code are sometimes published too.

It doesn't help. It just means outsiders get to read bad peer reviews and bad code, but the right people within the system don't actually care. There's no way for the public to hold anyone to account, science is a law unto itself because politicians are generally not scientifically minded. They can be easily bullied with technobabble. The few that aren't susceptible to this don't tend to rise to the top.

Re: How many medical studies are faked or flawed?

#227

Post by mike_hearn » Wed, Sep 20, 2023, 10:10 AM UTC

queuebert wrote: post #181

jjslocum3 wrote: post #151

> ...it would help if journals routinely asked authors to share their IPD Why couldn't a bad actor just fake the raw data? Isn't that what Climategate was all about?

That's hard to do such that it is undetectable to competent biostatisticians. Edit: See the Dan Ariely drama for an example.

We don't actually know it's hard to do because the base rate of fraud is unknown.

Re: How many medical studies are faked or flawed?

#228

Post by mike_hearn » Wed, Sep 20, 2023, 10:22 AM UTC

noslenwerdna wrote: post #137

bjourne wrote: post #22

We all know it is super-easy to cheat in science, but not much can be done about it. Short of requiring replication for every result published which isn't feasible if the study was costly to produce. And the problem isn't confined to medicine either. How many studies in hpc of the type "we setup this benchmark and our novel algorithm/implementation won!" aren't also faked or flawed?

Require experimental analysis to be pre-registered? That is how it's done in particle physics, and it works well.

It helps a lot but rests on the assumption that the people with whom you're pre-registering will check that you did the same thing you claimed you'd do. And also, that they won't tolerate gamed pre-registrations e.g. pre-registrations so vague that they admit many different post-hoc analyses.

Sometimes this is true. One example is when the people verifying the pre-registration work for a regulator. The introduction of pre-registration seems to be at least partly behind the collapse in pharma productivity:

https://journals.plos.org/plosone/article?id=10.1371/journal...

17 of 30 studies (57%) published prior to 2000 showed a significant benefit of intervention on the primary outcome in comparison to only 2 among the 25 (8%) trials published after 2000 (χ2=12.2,df= 1, p=0.0005). There has been no change in the proportion of trials that compared treatment to placebo versus active comparator. Industry co-sponsorship was unrelated to the probability of reporting a significant benefit. Pre-registration in clinicaltrials.gov was strongly associated with the trend toward null findings.

But when journals do this the results seem to be worse. They aren't strongly incentivized to improve anything, so you can get studies that claim to be pre-registered but do something different to what the pre-registration said they'd do.

Re: How many medical studies are faked or flawed?

#229

Post by ghomem » Wed, Sep 20, 2023, 10:23 AM UTC

junofan wrote: post #195

strken wrote: post #177

Earlier quoted context omitted.

"Trust the science" is dumb, but "trust that the single-most reviewed medical studies of the past decade with 30k+ participants aren't outright bullshit" is enough for me. The risk for those studies is probably closer to what happened with Astrazeneca, where a very rare side effect got missed because 30k people weren't enough to detect it. I think skepticism is healthy to a degree, but ironically, the more adversaria…

I mean I’m as vaccinated as anyone, but the originally reported 95% effectiveness against symptomatic COVID-19 with the monovalent vaccine just doesn’t seem realistic in hindsight. Who knows what complex set of incentives was at play. If you’re running a trial site and your business relies on continued contracts from Pfizer, are you reporting every cough, fever, or headache?

I think this was the original paper:

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7745181/

Sample size was 43k people split in treatment and control group. Adding it here for context.

Re: How many medical studies are faked or flawed?

#230

Post by mharig » Wed, Sep 20, 2023, 10:56 AM UTC

Llamamoe wrote: post #221

riperoni wrote: post #214

Earlier quoted context omitted.

Severe side effects or death cases were excluded from the Pfizer phase 3 study by conveniently shifting blame on patients, inventing diagnoses or claiming they had a Covid infection. Even if all that were not intentional, it would certainly raise the question why this should be the only case of fabricating data for studies. Since the RNA-vaccines are obviously not as positively effective as claimed, what else is ther…

Early into the pandemic, the original COVID strains had like 40% rate of permanent aftereffects, and the hospitalisation and death rates were insane. At the time, the vaccines have definitely saved lives, and the "covered up severe side effects or death" you talk about never came up in the real world, so...

No.