How many medical studies are faked or flawed? • Hacker News Board

Re: How many medical studies are faked or flawed?

#191

Post by jonlucc » Wed, Sep 20, 2023, 1:57 AM UTC

ancorevard wrote: post #114

onlyrealcuzzo wrote: post #96

Earlier quoted context omitted.

As someone who works in Data Science at FAANG - if you look hard enough - there is something questionably wrong in every step of the data funnel. And that's when I believe people do have a somewhat best effort to maximize profits. There are plenty of people that only care about career progression and think they can get away with lying and cheating their way to the top. They wouldn't believe that if it didn't work som…

"And that's when I believe people do have a somewhat best effort to maximize profits." Nope. I actually think that if you do scientific research as a company (profit) it may make you less bad/less likely to do fraud compared to academia (non-profit). Reason is that there are more ways to punish you, employees, board, investors, etc in a profit seeking vehicle, and as a profit seeking vehicle being caught must be part…

Completely anecdotally, I find this to be true. I do pre-clinical research (disclaimer: for a pharma company), and if you hand me a paper describing some research result in my field, it’s much more likely to reproduce in my hands if it’s from industry. I know from talking to my academic research friends that there are a ton of shortcuts, rotating inexperienced staff, and misaligned incentives. I’m not saying pharma is perfect, but in my experience, they’re more reproducible.

Fwiw, I do not know of any data in my realm which have been molded, cherry picked, intentionally misrepresented, falsified, or otherwise fake or flawed. I don’t work with clinical trial data, so if that were happening, it wouldn’t be on my desk.

Re: How many medical studies are faked or flawed?

#192

Post by ergonaught » Wed, Sep 20, 2023, 1:57 AM UTC

How many replies will originate from those who actually read the linked content, engage with it thoughtfully, and sidestep their own biases, distortions, prejudices, and ignorance?

An even lower percentage.

This species and its cultures are suffering an accelerating cognitive landslide.

Coupled with the many other accelerating descents, it's a shame, really.

I think we almost made it.

Almost.

Re: How many medical studies are faked or flawed?

#193

Post by robwwilliams » Wed, Sep 20, 2023, 1:58 AM UTC

BurningFrog wrote: post #63

Ilverin wrote: post #47

Earlier quoted context omitted.

Well you need peer reviewed papers in highish impact journals to get tenure. For some people, the only way they're getting that is by cheating.

Doing research has a lottery element. You explore something that might reveal an important discovery. And sometimes it just doesn't. That doesn't mean you're a bad scientist, just an unlucky one. But it does mean you can't get tenure. So it's easy to understand why people fake results to secure a career.

It is not all that difficult to select or find an interesting scientific questions for which a Yes, a No, or even a Maybe is publishable and interesting. Two on my highest impact studies have been fun “No” studies—-

1. No, there is minimal or no numerical matching between populations of neurons in retina (ganglion cells) and populations of principal neurons in their CNS target (the thalamus). That demolished the plausible/attractive numerical matching hypothesis. I was trying valiantly to support it ;-)

https://pubmed.ncbi.nlm.nih.gov/14657177/

2. No, there is no strong coupling of volumes of different brain regions due to “developmental constraints” in brain growth patterns. https://pubmed.ncbi.nlm.nih.gov/23011133/

That idea just struck me as silly from an evolutionary and comparative perspective. We were happy to call it into doubt.

I suspect many of the comments are being made by damn fine programmers who know right from wrong ;-) a la Dijkstra. But in biology and clinical research, defining right and wrong is an ill-defined problem with lots of barely tangible and invisible confounders.

We should still demand well designed, implemented, and analyzed experimental or observational data sets.

However, that alone is not nearly enough to ensure meaningful and generalizable results. The meta-analyses were supposed to help at this level for clinical trials but have been gamed by bad actors with career objective that don’t consider patient outcomes even a bit.

Highlighting the problem is a huge step forward and it looks like AI may provide some near-future help along with more complete data release requirements.

If you have done biology—- Hot. Wet. Mess. But beautiful.

Re: How many medical studies are faked or flawed?

#194

Post by jonlucc » Wed, Sep 20, 2023, 2:02 AM UTC

bee_rider wrote: post #79

onthecanposting wrote: post #28

Earlier quoted context omitted.

PIs may not be able to raise money for replication studies. Some of this is a consequence of guidelines for federal funds to prevent waste on duplication of effort.

This points to the real problem and also where the responsibility and interest ought to be to fix it. There’s no replication crisis for academics because they have a meatspace social network of academics; they go to conferences together and know each other. You can just ignore a paper if you know the author is an idiot. If medical studies are faked, is it a problem? Presumable regulatory agencies are using these stud…

At least in pre-clinical research, there’s also a mechanism by which regulatory agencies can sometimes force industry to pay for replication studies. It’s a little complicated, because sometimes there aren’t many groups who can reliably do the study, and clinical trials are so expensive and time-consuming to run, it would be difficult to do at that scale. Rodent studies are generally under $1M and monkey studies are generally under $5M (up a lot recently). Clinical trials are in the hundreds of millions of dollars and take years to run.

Re: How many medical studies are faked or flawed?

#195

Post by junofan » Wed, Sep 20, 2023, 2:04 AM UTC

strken wrote: post #177

avalys wrote: post #173

Earlier quoted context omitted.

Ah, but don’t worry folks, those vaccines which generated $40B in profits for Pfizer and Moderna, those are just A-ok. Trust the science, which in this case is perfect, unassailable and uncorruptible (unlike everything else the pharma industry does). Don’t question the faith.

"Trust the science" is dumb, but "trust that the single-most reviewed medical studies of the past decade with 30k+ participants aren't outright bullshit" is enough for me. The risk for those studies is probably closer to what happened with Astrazeneca, where a very rare side effect got missed because 30k people weren't enough to detect it. I think skepticism is healthy to a degree, but ironically, the more adversaria…

I mean I’m as vaccinated as anyone, but the originally reported 95% effectiveness against symptomatic COVID-19 with the monovalent vaccine just doesn’t seem realistic in hindsight. Who knows what complex set of incentives was at play. If you’re running a trial site and your business relies on continued contracts from Pfizer, are you reporting every cough, fever, or headache?

Re: How many medical studies are faked or flawed?

#196

Post by avalys » Wed, Sep 20, 2023, 2:08 AM UTC

strken wrote: post #177

avalys wrote: post #173

Earlier quoted context omitted.

Ah, but don’t worry folks, those vaccines which generated $40B in profits for Pfizer and Moderna, those are just A-ok. Trust the science, which in this case is perfect, unassailable and uncorruptible (unlike everything else the pharma industry does). Don’t question the faith.

"Trust the science" is dumb, but "trust that the single-most reviewed medical studies of the past decade with 30k+ participants aren't outright bullshit" is enough for me. The risk for those studies is probably closer to what happened with Astrazeneca, where a very rare side effect got missed because 30k people weren't enough to detect it. I think skepticism is healthy to a degree, but ironically, the more adversaria…

Peer review is not a magic spell that reveals the truth. “Most-reviewed” is meaningless here if the underlying data was not collected or reported properly.

Re: How many medical studies are faked or flawed?

#197

Post by frozenport » Wed, Sep 20, 2023, 2:17 AM UTC

wrp wrote: post #185

waterheater wrote: post #153

Earlier quoted context omitted.

Two relevant perspectives to share: "It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgment of trusted physicians or authoritative medical guidelines. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as an editor of the New England Journal of Medicine." -Marcia Angell "If the image of medicine I have conve…

Serving as editor of a major journal in any discipline can make one cynical. I read many years ago a quote from a mathematics editor that a large fraction of published proofs were faulty, and from a computer science editor that a large fraction of published theorems were false. (My foggy memory gives 1/2 and 1/3 for the fractions, but I don't remember which was which.)

Almost nobody has the throughput to check equations at all

Re: How many medical studies are faked or flawed?

#198

Post by jonlucc » Wed, Sep 20, 2023, 2:20 AM UTC

dukeofdoom wrote: post #176

Tijdreiziger wrote: post #171

Earlier quoted context omitted.

> The few times I visited a hospital I met people there with complications from previous drugs. Well… yeah? All medications have a list of possible side effects in the leaflet. It’s just that the disease tends to be worse.

Sorry, I'm no longer convinced of this. I think there's a large range of procedures/treatments being done that don't work, that kill people all the time. And in the end you are often just trading short term risk, for a longer payoff. Say you have prostate cancer. Should you risk the low chance of dying of prostate cancer 10 years from now, or the higher chance of incontinence, and medical complications from the treat…

That is why doctors are supposed to consent their patients and weigh those risks. For what it’s worth, the risk of incontinence form prostate cancer surgery has been dropping quickly as traditional surgery is replaced by robotic surgery, sparing the nerve that runs close under the prostate.

Re: How many medical studies are faked or flawed?

#199

Post by cm2012 » Wed, Sep 20, 2023, 2:20 AM UTC

autoexec wrote: post #127

paulusthe wrote: post #77

Earlier quoted context omitted.

It's more complex than just that. Sure, there's the people trying to make a dollar who are willing to do bad science in order to get the result they want. But there's also the general publication bias against replication studies - who wants to read them, and who wants to do them (they're not usually seen as prestigious academically: most academics want to test their ideas, not those of others. And then there's cultur…

> ut there's also the general publication bias against replication studies - who wants to read them, and who wants to do them (they're not usually seen as prestigious academically: most academics want to test their ideas, not those of others. Academia seems like the idea place for this. Why not require a certain number of replicated studies in order to get a degree? Universities could then be constantly churning out…

Maybe a school should start giving PhDs for good replications.

Re: How many medical studies are faked or flawed?

#200

Post by eru » Wed, Sep 20, 2023, 2:33 AM UTC

somenameforme wrote: post #17

"For more than 150 trials, Carlisle got access to anonymized individual participant data (IPD). By studying the IPD spreadsheets, he judged that 44% of these trials contained at least some flawed data: impossible statistics, incorrect calculations or duplicated numbers or figures, for instance. And in 26% of the papers had problems that were so widespread that the trial was impossible to trust, he judged — either bec…

> So, 70% fake/flawed.

It's worse. First, there's a selection bias in what trials the author can get data for. Second, the 44% + 26% are just those where the author can detect problems from reading their data alone. If someone convincingly fakes their data, the author can't detect that.