How many medical studies are faked or flawed? • Hacker News Board

Re: How many medical studies are faked or flawed?

#51

Post by sfink » Tue, Sep 19, 2023, 5:47 PM UTC

gmd63 wrote: post #41

bjourne wrote: post #22

We all know it is super-easy to cheat in science, but not much can be done about it. Short of requiring replication for every result published which isn't feasible if the study was costly to produce. And the problem isn't confined to medicine either. How many studies in hpc of the type "we setup this benchmark and our novel algorithm/implementation won!" aren't also faked or flawed?

Plenty can be done about it. We can start by increasing the standards for experimental proof past words on a piece of paper. Stream or record your experiment. If a 12 year old has the ability to stream his gaming life on twitch, scientists should be able to record what they are doing in more detail. You could start a new journal with higher level of prestige that only publishes experiments that adhere to more modern…

> We can start by increasing the standards for experimental proof past words on a piece of paper.

That is overly cynical. We are already well past that standard.

We require words on a piece of paper that were written by somebody from a recognizable institution, or by an AI operated by somebody from a recognizable institution.

Re: How many medical studies are faked or flawed?

#52

Post by lurquer » Tue, Sep 19, 2023, 5:48 PM UTC

goosinmouse wrote: post #24

denimnerd42 wrote: post #15

Earlier quoted context omitted.

Furan content in coffee too. Very low amount but still. Coffee prevents headaches for me so I'll always drink it. And no it's not related to physical dependence although at this point the withdrawal will guarantee a headache.

I can relate. I never drank anything with caffeine and would get headaches fairly often. Headaches were never too bad or too often to need medical attention but was just normal part of life. I started drinking coffee on road trips and drives over 3 hours long and noticed that my headache coming on would go away right after. Now i drink coffee twice daily and i'll get a headache once a month at most.

I drank five or six cups of coffee a day for decades. I’d even have a cup before going to bed — that’s how tolerant I had become.

Got a mild flu/Covid/cold couple years ago. Better in a week. But, during the illness and since, the slightest bit of caffeine would make be incredibly wired to the point of panic attacks. Had to quit cold turkey. I’ve tried a cup now and again, and it’s the same thing: 6 hours of overwhelming anxiety.

Wierd. It’s like I became hypersensitive to caffeine. Oddly, though, nicotine doesn’t have that effect, and I always figured the two stimulants were similar.

Re: How many medical studies are faked or flawed?

#53

Post by waihtis » Tue, Sep 19, 2023, 5:48 PM UTC

PaulHoule wrote: post #9

waihtis wrote: post #5

Earlier quoted context omitted.

why coffee and red wine specifically?

Both of those are substances that should be harmful based on the effects of the major ingredients: e.g. caffeine is addictive and when I am deprived of it my schizotypy flares up and I have paranoid episodes (I yell at my wife “why the hell are you always standing where I want to go next?”) I have had two doctors tell me three other reasons why I should quit. Look in the medical literature and it seems outright spamm…

Yes but that's not something you can automatically draw inferences from. Exercise is harmful to you in a short enough time interval but benefits you on the long run.

Re: How many medical studies are faked or flawed?

#54

Post by Zalastax » Tue, Sep 19, 2023, 5:50 PM UTC

somenameforme wrote: post #17

"For more than 150 trials, Carlisle got access to anonymized individual participant data (IPD). By studying the IPD spreadsheets, he judged that 44% of these trials contained at least some flawed data: impossible statistics, incorrect calculations or duplicated numbers or figures, for instance. And in 26% of the papers had problems that were so widespread that the trial was impossible to trust, he judged — either bec…

Is there not full or at least partial overlap between the 44 % and the 26 %? Which would mean not 70 % but some smaller number?

Re: How many medical studies are faked or flawed?

#55

Post by waihtis » Tue, Sep 19, 2023, 5:50 PM UTC

lurquer wrote: post #52

goosinmouse wrote: post #24

Earlier quoted context omitted.

I can relate. I never drank anything with caffeine and would get headaches fairly often. Headaches were never too bad or too often to need medical attention but was just normal part of life. I started drinking coffee on road trips and drives over 3 hours long and noticed that my headache coming on would go away right after. Now i drink coffee twice daily and i'll get a headache once a month at most.

I drank five or six cups of coffee a day for decades. I’d even have a cup before going to bed — that’s how tolerant I had become. Got a mild flu/Covid/cold couple years ago. Better in a week. But, during the illness and since, the slightest bit of caffeine would make be incredibly wired to the point of panic attacks. Had to quit cold turkey. I’ve tried a cup now and again, and it’s the same thing: 6 hours of overwhel…

Try yerba mate, like seriously. It gives you a very smooth caffeine-induced motivation boost without any of the anxiety effects.

I frequently switch between that and coffee (coffee has a much more pronounced effect and sometimes you have to grind)

Re: How many medical studies are faked or flawed?

#56

Post by tppiotrowski » Tue, Sep 19, 2023, 5:50 PM UTC

some_random wrote: post #18

While we're on the subject, how many studies period are faked or flawed to the point of being useless? It seems to me that the scientific community's reaction to the replication crisis has been to ignore it.

> While we're on the subject, how many studies period are faked or flawed to the point of being useless?

Or how many studies are useless, period? It's like publishing a memoir to Amazon. You can now say "author" on your resume, or when you're introduced or at cocktail parties but nobody finds any value in what you have to say. You can also use ChatGPT because people might not notice.

Re: How many medical studies are faked or flawed?

#57

Post by morelisp » Tue, Sep 19, 2023, 5:50 PM UTC

somenameforme wrote: post #17

"For more than 150 trials, Carlisle got access to anonymized individual participant data (IPD). By studying the IPD spreadsheets, he judged that 44% of these trials contained at least some flawed data: impossible statistics, incorrect calculations or duplicated numbers or figures, for instance. And in 26% of the papers had problems that were so widespread that the trial was impossible to trust, he judged — either bec…

> The finding falls in line with other large scale replication studies in medicine, which have had replication success rates ranging from 11 to 44%.

These numbers seem almost wholly unrelated. A perfectly good study may be extremely difficult to replicate (or even the original purpose of replication - the experiment as described in the paper may simply not be sufficient); and an attempt at replication (or refutation), successful or not, is under the same pressure to be faked or flawed as the original paper.

Re: How many medical studies are faked or flawed?

#58

Post by michaelrpeskin » Tue, Sep 19, 2023, 5:52 PM UTC

bjourne wrote: post #22

We all know it is super-easy to cheat in science, but not much can be done about it. Short of requiring replication for every result published which isn't feasible if the study was costly to produce. And the problem isn't confined to medicine either. How many studies in hpc of the type "we setup this benchmark and our novel algorithm/implementation won!" aren't also faked or flawed?

The "setup this benchmark" happens in medicine all of the time, and it's super insidious and no one sees it. It's hard to tease out.

For example, say you have a new (patentable) drug that you're trying to get approved and replace an old (generic, cheap) drug. You need to prove that the new drug is at least as safe as the old one and/or more effective than the old one. Which sounds reasonable.

Let's say that the old drug is super safe and that the new one may not be. What you can do is set up your RCT so that you surreptitiously underdose the control arm so the safe drug looks less effective. And then you can say that your new drug is more effective than the old one.

Peer review doesn't notice this because you can easily hide it in the narrative of the methods section. I've seen it a couple of times.

So you can easily have "gold standard RCTs" that get the result you want just by subtle changes in the study design.

Re: How many medical studies are faked or flawed?

#59

Post by qt31415926 » Tue, Sep 19, 2023, 5:52 PM UTC

somenameforme wrote: post #17

"For more than 150 trials, Carlisle got access to anonymized individual participant data (IPD). By studying the IPD spreadsheets, he judged that 44% of these trials contained at least some flawed data: impossible statistics, incorrect calculations or duplicated numbers or figures, for instance. And in 26% of the papers had problems that were so widespread that the trial was impossible to trust, he judged — either bec…

I don't think the statement reads that the 44% and the 26% should be additive. Especially given the zombie graphic where it looks like they overlap the 26 on top of the 44, where the orange bar is the 26% and the remaining yellow bar is the 44%

Re: How many medical studies are faked or flawed?

#60

Post by colechristensen » Tue, Sep 19, 2023, 5:55 PM UTC

RugnirViking wrote: post #46

some_random wrote: post #18

While we're on the subject, how many studies period are faked or flawed to the point of being useless? It seems to me that the scientific community's reaction to the replication crisis has been to ignore it.

> the scientific community's reaction to the replication crisis has been to ignore it What can they do? It's an incredibly hard problem to solve. It's like asking why the buisness community has done nothing to adress the housing crisis. Large scale culture changes, or the entire structure of the way science is conducted and funded, would be the only solutions

If a university was actually in the business of seeking truth, there is much they could do to solve this.

* working groups "red teams" whose whole job it is to find weaknesses in papers published at the university

* post mortems after finding papers with serious flaws exposing the problem and coming up with constructive corrective actions

* funding / forcing researchers to devote a certain amount of time to replicating significant results

* working groups of experts in statistics and study design available to consult

* systems to register studies, methodologies, data sets, etc. with the aim of eliminating error and preventing post-hoc fishing expeditions for results

The whole-ass purpose of a university is seeking knowledge. They are fully capable of doing a better job of it but they don't because what they actually focus on are things like fundraising, professional sports teams, constructing attractive buildings, and advancing ranking.

Most universities would be better off just firing and not replacing 90% of their administration.