Live data from Hacker News

How many medical studies are faked or flawed?

web.archive.org

81–90 of 257 posts

Re: How many medical studies are faked or flawed?

#81

Those are two wildly different things. Most things are flawed. Deliberately fabricating results is anathema to advancing science.

True that, but the article does distinguish between "flawed" (44%) and "problems that were so widespread that the trial was impossible to trust" (26%).

Really, though, how/why would we expect otherwise? There's nothing in the system to prevent it, and plenty to incentivize it. There's really no good reason to expect (under the current system) that it would not happen (a lot). Without systemic change, it will not get better.

Re: How many medical studies are faked or flawed?

#82
post #11

Pretty sure most studies about the effects of coffee and red wine are flawed.

What!? Everyone knows a glass of wine once a day is good for your heart, but also even a single glass a year is way too much and causes irreversible damage ;) The old adage applies: "everything nice is either illegal or bad for your health". Or both, I would add.

I think that something can have both good and bad effects.

Re: How many medical studies are faked or flawed?

#83

"For more than 150 trials, Carlisle got access to anonymized individual participant data (IPD). By studying the IPD spreadsheets, he judged that 44% of these trials contained at least some flawed data: impossible statistics, incorrect calculations or duplicated numbers or figures, for instance. And in 26% of the papers had problems that were so widespread that the trial was impossible to trust, he judged — either bec…

You don't need "big pharma" for this. Researchers at universities also do these kinds of things because it helps them advance their careers.

> Researchers at universities also do these kinds of things because it helps them advance their careers.

This is a huge problem and in my opinion is mostly due to bad incentive structures and bad statistical/methodological education. I'm sure there are plenty of cases where there is intentional or at least known malpractice, but I would argue that most bad research is done in good faith.

When I was working on a PhD in biostatistics with a focus on causal inference among other things, I frequently helped out friends in other departments with data analysis. More often than not, people were working with sample sizes that are too small to provide enough power to answer their questions, or questions that simply could not be answered by their study design. (e.g. answering causal questions from observational data*).

In once instance, a friend in an environmental science program had data from an experiment she conducted where she failed to find evidence to support her primary hypothesis. It's nearly impossible to publish null results, and she didn't have funding to collect more data and had to get a paper out of it.

She wound up doing textbook p-hacking; testing a ton of post-hoc hypotheses on subsets of data. I tried to reel things back but I couldn't convince her to not continue because "that's how they do things" in her field. In reality she didn't really have a choice if she wanted to make progress towards her degree. She was a very smart person, and p-hacking is conceptually not hard to understand, but she was incentivized to not understand it or to not look at her research in that way.

* Research in causal inference is mostly about rigorously defining the (untestable) causal assumptions you must make and developing methods to answer causal questions from observational data. Even if an argument can be made that you can make those assumptions in a particular case, there is another layer of modeling assumptions you'll end up making depending on the method you're using. In my experience it's pretty rare that you can really have much confidence that your conclusions about a causal question if you can't run a real experiment.

Re: How many medical studies are faked or flawed?

#84
post #11

Earlier quoted context omitted.

What!? Everyone knows a glass of wine once a day is good for your heart, but also even a single glass a year is way too much and causes irreversible damage ;) The old adage applies: "everything nice is either illegal or bad for your health". Or both, I would add.

I think that something can have both good and bad effects.

Yes, of course. But have you read journalists reporting on science findings? It's always an extreme "scientists now claim one glass of wine a day is good for you!" and "scientists discovered that even one glass of wine will ruin your life forever".

Never the middle ground, it's always a shocking new finding "by science" (spoiler: scientists seldom say the things newspapers and pop-science/nutrition & health articles claim they say).

Re: How many medical studies are faked or flawed?

#85
post #33
post #11

Earlier quoted context omitted.

What!? Everyone knows a glass of wine once a day is good for your heart, but also even a single glass a year is way too much and causes irreversible damage ;) The old adage applies: "everything nice is either illegal or bad for your health". Or both, I would add.

I'm nice so your point doesn't stand. Or does it ? I'm not illegal at least.

Well, you would say so even if you were illegal!

Re: How many medical studies are faked or flawed?

#86

Earlier quoted context omitted.

My free range chickens are protected by free range fox predators that are communal with them. Sometimes they will eat a chicken or two, but mostly they let the chickens be because they keep the free range parasites in check.

I definitely count on stinging insects that live in holes in my house (or that make holes) to keep other stinging insects away. What I've been told is that opossums are much worse than foxes in the sense that a fox will usually eat a chicken or two to survive but opossums seem to freak out and will kill all the hens in a henhouse in one go. I've often wished I could talk with my cats but more than ever I wish I could…

I have to say that I find foxes beautiful. Then again, I'm not a farmer nor do I keep chickens. I'm a city boy.

Re: How many medical studies are faked or flawed?

#87

While we're on the subject, how many studies period are faked or flawed to the point of being useless? It seems to me that the scientific community's reaction to the replication crisis has been to ignore it.

> It seems to me that the scientific community's reaction to the replication crisis has been to ignore it. They've always known so there hasn't actually been any new information from which to spur action. In the academic circles I've run in there has always been a strong mistrust of reported results and procedures based on past difficulties with internal efforts replicating results. Basically a right of passage for a…

That's the correct answer, but what is baffling is that this is news to so many people frequenting this website. Most everyone posting here has been though a university, but hardly anyone has been involved in the pursuit of research.

Re: How many medical studies are faked or flawed?

#88
The few times I visited a hospital I met people there with complications from previous drugs. One with liver failure. Thanks to that experience no covid shot for me, plus I actually read the pfzier study the judge forced them to release. I now view drug companies on the level of Cartels in Mexico with politicians in their pockets.

Re: How many medical studies are faked or flawed?

#89

While we're on the subject, how many studies period are faked or flawed to the point of being useless? It seems to me that the scientific community's reaction to the replication crisis has been to ignore it.

> the scientific community's reaction to the replication crisis has been to ignore it What can they do? It's an incredibly hard problem to solve. It's like asking why the buisness community has done nothing to adress the housing crisis. Large scale culture changes, or the entire structure of the way science is conducted and funded, would be the only solutions

It doesn’t have to be large scale. It can start with a single journal, with a single university, with a single scientist.

Make peer review public. Weight replication studies more. Make conducted peer reviews and replications an important metric for tenure and grants. Publish data and code alongside papers.

Re: How many medical studies are faked or flawed?

#90
post #5

Earlier quoted context omitted.

why coffee and red wine specifically?

Both of those are substances that should be harmful based on the effects of the major ingredients: e.g. caffeine is addictive and when I am deprived of it my schizotypy flares up and I have paranoid episodes (I yell at my wife “why the hell are you always standing where I want to go next?”) I have had two doctors tell me three other reasons why I should quit. Look in the medical literature and it seems outright spamm…

>caffeine is addictive and when I am deprived of it my schizotypy flares up and I have paranoid episodes

Dose size matters.

Post reply on HN