Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How many medical studies are faked or flawed?

web.archive.org

Post reply on HN
Print view View original

71–80 of 257 posts

Re: How many medical studies are faked or flawed?

#71

Post by rqtwteye »

adamsb6 wrote: post #66
some_random wrote: post #18

While we're on the subject, how many studies period are faked or flawed to the point of being useless? It seems to me that the scientific community's reaction to the replication crisis has been to ignore it.

The replication crisis is even a thing because of very strong career incentives. You might think that publishing about the replication crisis itself would be great for your career, but perhaps not. Maybe the incentives to be able to bullshit your way to a professorship are so great that no one wants to rock the boat.

"Maybe the incentives to be able to bullshit your way to a professorship are so great that no one wants to rock the boat."

Our whole economy is fueled by people bullshitting each other.

Re: How many medical studies are faked or flawed?

#72

Post by adasdasdas »

somenameforme wrote: post #17

"For more than 150 trials, Carlisle got access to anonymized individual participant data (IPD). By studying the IPD spreadsheets, he judged that 44% of these trials contained at least some flawed data: impossible statistics, incorrect calculations or duplicated numbers or figures, for instance. And in 26% of the papers had problems that were so widespread that the trial was impossible to trust, he judged — either bec…

I also see similar findings in tech where most experiment results are "fudged". In some cases, people run the "same" experiment 5+ times until one is stat sig.

Re: How many medical studies are faked or flawed?

#73

Post by LMYahooTFY »

cypress66 wrote: post #23
somenameforme wrote: post #17

"For more than 150 trials, Carlisle got access to anonymized individual participant data (IPD). By studying the IPD spreadsheets, he judged that 44% of these trials contained at least some flawed data: impossible statistics, incorrect calculations or duplicated numbers or figures, for instance. And in 26% of the papers had problems that were so widespread that the trial was impossible to trust, he judged — either bec…

You don't need "big pharma" for this. Researchers at universities also do these kinds of things because it helps them advance their careers.

What's your point? We have "big pharma" for this.

Do you think CocaCola and the Sacklers had their own unique ideas shared by no one else? That we've filtered all scrupulous people out of industry?

Scruples are an abstraction at that scale.

Re: How many medical studies are faked or flawed?

#74

Post by gmd63 »

gustavus wrote: post #50
gmd63 wrote: post #41

Earlier quoted context omitted.

Plenty can be done about it. We can start by increasing the standards for experimental proof past words on a piece of paper. Stream or record your experiment. If a 12 year old has the ability to stream his gaming life on twitch, scientists should be able to record what they are doing in more detail. You could start a new journal with higher level of prestige that only publishes experiments that adhere to more modern…

This seems overly simplistic, I mean part of the reason so many studies are so difficult is because they can happen over the period of months or years. This isn't like your High School chemistry experiment. These studies can be massive longitudinal studies that take years to conclude and work on. If the answers were easy I'm sure someone would've implemented it already, it turns out those kind of things are hard. I'd…

I'm interested more in why it specifically would be hard to just show people what you are doing (or did) instead of telling them a story at the end of months or years.

Critical details and oversights can be lost in translation between reality and LaTeX that could be easily pointed out by third party observers.

Re: How many medical studies are faked or flawed?

#75

Post by Clubber »

the_af wrote: post #11
ketanmaheshwari wrote: post #4

Pretty sure most studies about the effects of coffee and red wine are flawed.

What!? Everyone knows a glass of wine once a day is good for your heart, but also even a single glass a year is way too much and causes irreversible damage ;) The old adage applies: "everything nice is either illegal or bad for your health". Or both, I would add.

Or will produce a baby.

Re: How many medical studies are faked or flawed?

#76

Post by tarxvf »

sampo wrote: post #19
the_af wrote: post #11

Earlier quoted context omitted.

What!? Everyone knows a glass of wine once a day is good for your heart, but also even a single glass a year is way too much and causes irreversible damage ;) The old adage applies: "everything nice is either illegal or bad for your health". Or both, I would add.

> "everything nice is either illegal or bad for your health" And is known to cause cancer in the State of California.

Well that's alright then, I'm in New Hampshire.

Re: How many medical studies are faked or flawed?

#77

Post by paulusthe »

somenameforme wrote: post #17

"For more than 150 trials, Carlisle got access to anonymized individual participant data (IPD). By studying the IPD spreadsheets, he judged that 44% of these trials contained at least some flawed data: impossible statistics, incorrect calculations or duplicated numbers or figures, for instance. And in 26% of the papers had problems that were so widespread that the trial was impossible to trust, he judged — either bec…

It's more complex than just that. Sure, there's the people trying to make a dollar who are willing to do bad science in order to get the result they want. But there's also the general publication bias against replication studies - who wants to read them, and who wants to do them (they're not usually seen as prestigious academically: most academics want to test their ideas, not those of others.

And then there's cultural differences in which people sometimes see a negative result as a "failure", don't publish it as a result, and instead skew the data and lie their asses off in order to gain prestige in their career. As long as nobody double checks you, you're good.

Re: How many medical studies are faked or flawed?

#78

Post by PaulHoule »

cratermoon wrote: post #30
the_af wrote: post #14

Earlier quoted context omitted.

Well, your free range chickens aren't truly free range if they don't have to fend off the occasional (and free range) fox! Nothing is more natural and free range than the occasional murder between animals.

My free range chickens are protected by free range fox predators that are communal with them. Sometimes they will eat a chicken or two, but mostly they let the chickens be because they keep the free range parasites in check.

I definitely count on stinging insects that live in holes in my house (or that make holes) to keep other stinging insects away.

What I've been told is that opossums are much worse than foxes in the sense that a fox will usually eat a chicken or two to survive but opossums seem to freak out and will kill all the hens in a henhouse in one go.

I've often wished I could talk with my cats but more than ever I wish I could ask them what they knew about the fox. There is this lady

https://www.youtube.com/@debs3289

who meets them in the street, has them come to her door, and feeds them chicken (!) Secretly I imagine that the fox is really a

https://en.wikipedia.org/wiki/Kitsune

and my wife is always reminding me that it has just one tail, not nine.

Re: How many medical studies are faked or flawed?

#79

Post by bee_rider »

onthecanposting wrote: post #28
some_random wrote: post #18

While we're on the subject, how many studies period are faked or flawed to the point of being useless? It seems to me that the scientific community's reaction to the replication crisis has been to ignore it.

PIs may not be able to raise money for replication studies. Some of this is a consequence of guidelines for federal funds to prevent waste on duplication of effort.

This points to the real problem and also where the responsibility and interest ought to be to fix it.

There’s no replication crisis for academics because they have a meatspace social network of academics; they go to conferences together and know each other. You can just ignore a paper if you know the author is an idiot.

If medical studies are faked, is it a problem? Presumable regulatory agencies are using these studies or something, right? Looks like the FDA and NSF need to fund some more replication studies.

Re: How many medical studies are faked or flawed?

#80

Post by costigan »

somenameforme wrote: post #17

"For more than 150 trials, Carlisle got access to anonymized individual participant data (IPD). By studying the IPD spreadsheets, he judged that 44% of these trials contained at least some flawed data: impossible statistics, incorrect calculations or duplicated numbers or figures, for instance. And in 26% of the papers had problems that were so widespread that the trial was impossible to trust, he judged — either bec…

I think you misread the article slightly. The 26% of zombie papers are inside the 44% of papers that contained at least some flawed data. Look at the figure just below this quote where the blue bar, indicating papers he thought were ok, covers more than 50%.
Post reply on HN