> ...it would help if journals routinely asked authors to share their IPD
Why couldn't a bad actor just fake the raw data? Isn't that what Climategate was all about?
How many medical studies are faked or flawed?
151–160 of 257 posts
Re: How many medical studies are faked or flawed?
#152Earlier quoted context omitted.
> That doesn't mean you're a bad scientist, just an unlucky one. But it does mean you can't get tenure. That sounds like a really easy problem to solve. Just treat valid science as important regardless of the results. The results shouldn't matter unless they've been replicated and verified anyway.
Too easy. Define "valid" though.
Valid as in meaningfully peer reviewed to avoid flawed/badly designed studies as well as total garbage (for example https://nerdist.com/article/fake-star-wars-midi-chlorian-pap...) but the gold standard should be replication.
We should reward quality work, not simply the number of research papers (since it's easy to churn out trash) or what the results are (because until they are verified they could be faked).
Re: How many medical studies are faked or flawed?
#153"For more than 150 trials, Carlisle got access to anonymized individual participant data (IPD). By studying the IPD spreadsheets, he judged that 44% of these trials contained at least some flawed data: impossible statistics, incorrect calculations or duplicated numbers or figures, for instance. And in 26% of the papers had problems that were so widespread that the trial was impossible to trust, he judged — either bec…
Two relevant perspectives to share:
"It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgment of trusted physicians or authoritative medical guidelines. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as an editor of the New England Journal of Medicine." -Marcia Angell
"If the image of medicine I have conveyed is one wherein medicine lurches along, riven by internal professional power struggles, impelled this way and that by arbitrary economic and sociopolitical forces, and sustained by bodies of myth and rhetoric that are elaborated in response to major threats to its survival, then that is the image supported by this study." -Evelleen Richards, "Vitamin C and Cancer: Medicine or Politics?"
Re: How many medical studies are faked or flawed?
#154While we're on the subject, how many studies period are faked or flawed to the point of being useless? It seems to me that the scientific community's reaction to the replication crisis has been to ignore it.
> the scientific community's reaction to the replication crisis has been to ignore it What can they do? It's an incredibly hard problem to solve. It's like asking why the buisness community has done nothing to adress the housing crisis. Large scale culture changes, or the entire structure of the way science is conducted and funded, would be the only solutions
>What can they do?
There's literally thousands of different things they could do.
But why do anything when the real business of academia is in the tax-free hedge funds they operate and the government-subsidized international student gravy train? There's no short-term incentive to change anything.
Re: How many medical studies are faked or flawed?
#155"For more than 150 trials, Carlisle got access to anonymized individual participant data (IPD). By studying the IPD spreadsheets, he judged that 44% of these trials contained at least some flawed data: impossible statistics, incorrect calculations or duplicated numbers or figures, for instance. And in 26% of the papers had problems that were so widespread that the trial was impossible to trust, he judged — either bec…
As someone who works in Data Science at FAANG - if you look hard enough - there is something questionably wrong in every step of the data funnel. And that's when I believe people do have a somewhat best effort to maximize profits. There are plenty of people that only care about career progression and think they can get away with lying and cheating their way to the top. They wouldn't believe that if it didn't work som…
>Actually, I'd feel better if someone could convince me I'm completely unfounded here.
Unfortunately, your skepticism is not unfounded. Those in the industry conclude the same. Take, for instance, the editor in chief of The Lancet:
>The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness.
https://www.thelancet.com/journals/lancet/article/PIIS0140-6...
Re: How many medical studies are faked or flawed?
#156"For more than 150 trials, Carlisle got access to anonymized individual participant data (IPD). By studying the IPD spreadsheets, he judged that 44% of these trials contained at least some flawed data: impossible statistics, incorrect calculations or duplicated numbers or figures, for instance. And in 26% of the papers had problems that were so widespread that the trial was impossible to trust, he judged — either bec…
As someone who works in Data Science at FAANG - if you look hard enough - there is something questionably wrong in every step of the data funnel. And that's when I believe people do have a somewhat best effort to maximize profits. There are plenty of people that only care about career progression and think they can get away with lying and cheating their way to the top. They wouldn't believe that if it didn't work som…
I’ve literally never seen data-driven business decisions that weren’t using fatally flawed datasets or methods so bad that you’d be a fool to believe you were getting anything but gibberish out of them, except in trivial cases.
You quickly learn not to be the guy pointing out the problem that means we’ll need several people and months or years to gather and analyze data that would allow them to (maybe) support or disprove their conclusion, though. Nobody wants to hear it… because they don’t actually care, they just want to present themselves as doing data-driven decision making, for reasons of ego or for (personal, or company) marketing. It’s all gut feelings and big personalities pushing companies this way and that, once you cut through the pretend-science shit.
“Yeah, that graph looks great (soul dies a little) let’s do it”
Re: How many medical studies are faked or flawed?
#157[deleted]
Re: How many medical studies are faked or flawed?
#158Earlier quoted context omitted.
A lot who stay purely in Academia do it as well, presumably for the prestige.
Well you need peer reviewed papers in highish impact journals to get tenure. For some people, the only way they're getting that is by cheating.
It's worse than that: if enough people cheat, even the best people can't make the grade without cheating.
Re: How many medical studies are faked or flawed?
#159You know how (to use a familiar field) most of software development has become going through the motions, churning massive insecure bulk, doing nonsense rituals and making up myths about process and productivity, hopping jobs frequently, cutting corners and breaking rules, etc., and the purpose is usually not to produce trustworthy solutions, but only to further career and livelihood/wealth? With everything we've bee…
[deleted]
Re: How many medical studies are faked or flawed?
#160Pretty sure most studies about the effects of coffee and red wine are flawed.
[deleted]