> the scientific community's reaction to the replication crisis has been to ignore it What can they do? It's an incredibly hard problem to solve. It's like asking why the buisness community has done nothing to adress the housing crisis. Large scale culture changes, or the entire structure of the way science is conducted and funded, would be the only solutions
If a university was actually in the business of seeking truth, there is much they could do to solve this. * working groups "red teams" whose whole job it is to find weaknesses in papers published at the university * post mortems after finding papers with serious flaws exposing the problem and coming up with constructive corrective actions * funding / forcing researchers to devote a certain amount of time to replicati…
Universities can't do that, because they lack the expertise. Most of the time, the only people in a university capable of judging a study are the people who did it.
That's because universities are in the business of teaching. Apart from a few rare exceptions, universities don't have the money to hire redundant people. Instead of hiring many experts in the same topic, they prefer hiring a wider range of expertise, in order to provide better learning opportunities for the students.
"For more than 150 trials, Carlisle got access to anonymized individual participant data (IPD). By studying the IPD spreadsheets, he judged that 44% of these trials contained at least some flawed data: impossible statistics, incorrect calculations or duplicated numbers or figures, for instance. And in 26% of the papers had problems that were so widespread that the trial was impossible to trust, he judged — either bec…
As someone who works in Data Science at FAANG - if you look hard enough - there is something questionably wrong in every step of the data funnel. And that's when I believe people do have a somewhat best effort to maximize profits. There are plenty of people that only care about career progression and think they can get away with lying and cheating their way to the top. They wouldn't believe that if it didn't work som…
I don't think that's skeptical.
I do think there is an even worse issue - which is funding. The money incentive means you can fund studies that support whatever you want.
I'm not nearly qualified to make this argument, but has anyone ever suggested that we collectively do away with the principle that one must publish original research in order to receive a PhD? Maybe in something like entomology, there are enough undescribed beetle species out there to supply myriads of dissertations, but in other fields it seems like you are just incentivizing trivial, useless, or fraudulent research…
That wouldn't fix anything at all. You still get that PhD to do research and publish. The only fix for this is funded and career-advancing randomized replication. We should lean harder into the scientific method, not withdraw from it. Absolutely nothing else will work.
"For more than 150 trials, Carlisle got access to anonymized individual participant data (IPD). By studying the IPD spreadsheets, he judged that 44% of these trials contained at least some flawed data: impossible statistics, incorrect calculations or duplicated numbers or figures, for instance. And in 26% of the papers had problems that were so widespread that the trial was impossible to trust, he judged — either bec…
As someone who works in Data Science at FAANG - if you look hard enough - there is something questionably wrong in every step of the data funnel. And that's when I believe people do have a somewhat best effort to maximize profits. There are plenty of people that only care about career progression and think they can get away with lying and cheating their way to the top. They wouldn't believe that if it didn't work som…
"And that's when I believe people do have a somewhat best effort to maximize profits."
Nope. I actually think that if you do scientific research as a company (profit) it may make you less bad/less likely to do fraud compared to academia (non-profit).
Reason is that there are more ways to punish you, employees, board, investors, etc in a profit seeking vehicle, and as a profit seeking vehicle being caught must be part of the profit seeking calculation – in the end, the world of reality/physics will weigh your contribution.
I believe there is evidence that there is more fraudulent scientific research happening in non-profit vehicles/academia. Take for example an area where there are fewer profit seeking companies participating - social sciences. It's dominated by academia. Now look at the replication rate of social sciences.
I'm not nearly qualified to make this argument, but has anyone ever suggested that we collectively do away with the principle that one must publish original research in order to receive a PhD? Maybe in something like entomology, there are enough undescribed beetle species out there to supply myriads of dissertations, but in other fields it seems like you are just incentivizing trivial, useless, or fraudulent research…
As a completely unqualified laymen my initial question would be "If you don't reward novelty why would anyone focused on career building want to be novel?" Not that I doubt academia has people who want to push the cutting edge forward (if anything that seems to be the only reason to go into academia vs. private industry usually) but if I was a fresh-faced PhD aspirant I'd want to take the most reliable route to getting my degree and treading ground someone else has already walked seems like a much safer way to do that than novel research if the reward at the end is the same.
But maybe that's a good thing? I can't actually say a reason I think it'd be that terrible except for the profs doing novel research that would lose their some of their student workforce.
"For more than 150 trials, Carlisle got access to anonymized individual participant data (IPD). By studying the IPD spreadsheets, he judged that 44% of these trials contained at least some flawed data: impossible statistics, incorrect calculations or duplicated numbers or figures, for instance. And in 26% of the papers had problems that were so widespread that the trial was impossible to trust, he judged — either bec…
As someone who works in Data Science at FAANG - if you look hard enough - there is something questionably wrong in every step of the data funnel. And that's when I believe people do have a somewhat best effort to maximize profits. There are plenty of people that only care about career progression and think they can get away with lying and cheating their way to the top. They wouldn't believe that if it didn't work som…
Objectivity and honesty can be hard to find if all someone cares about is their reputation as a competent researcher or climbing the ladder. What do you think a potential solution would be for this? I feel like even in my own experience, trying something out and it not working feels like failure, when in fact to proclaim it a success or "fix" it is truly what harms both the endeavor for truth and the people reliant on the outcomes of these surveys
"For more than 150 trials, Carlisle got access to anonymized individual participant data (IPD). By studying the IPD spreadsheets, he judged that 44% of these trials contained at least some flawed data: impossible statistics, incorrect calculations or duplicated numbers or figures, for instance. And in 26% of the papers had problems that were so widespread that the trial was impossible to trust, he judged — either bec…
> Carlisle got access to anonymized individual participant data (IPD) I'm not in the industry so my question might have an obvious answer to those of you who are: How would one go about getting IPD if you wanted to run your own analysis of trial data or other data-driven research?
You'll need to reach out to the study authors with a request. If they are interested (you're going to publish something noteworthy with a citation for them (low chance), you want to bring them in on some funded research (better chance), etc) then they'll push it to their Institutional Review Board (a group of usually faculty and sometimes administrative staff at a University/Hospital/Org) who will review the request, the conditions of the initial data collection, legal restrictions, and then decide if they'll proceed with setting up some sort of IRB agreement / data use agreement. Unless you're a tenured professor somewhere or a respected researcher with some outside group then you probably won't get past any of those steps. Even allegedly anonymized data comes with the risk of exposure (and real penalties) not to mention the administrative overhead (expense, time, attention) that you'll need to be able to cover the cost of through some funded research. That research, btw, will also need to be through some IRB structure. You can tap a private firm that acts as an IRB but that's another process entirely and most certainly requires fat stacks of cash. Legal privacy concerns, ethical concerns, careerism (nobody wants you to find the 'carry the two' you forgot so you can crash their career prospects), bloated expenses (somebody has to pay for all of that paperwork, all those IRB salaries, etc) and etc, etc, etc all keep reproducibility of individual data frozen. Even within the same institution. Within the same team! You have to tread lightly with reproduction.
As someone who works in Data Science at FAANG - if you look hard enough - there is something questionably wrong in every step of the data funnel. And that's when I believe people do have a somewhat best effort to maximize profits. There are plenty of people that only care about career progression and think they can get away with lying and cheating their way to the top. They wouldn't believe that if it didn't work som…
"And that's when I believe people do have a somewhat best effort to maximize profits." Nope. I actually think that if you do scientific research as a company (profit) it may make you less bad/less likely to do fraud compared to academia (non-profit). Reason is that there are more ways to punish you, employees, board, investors, etc in a profit seeking vehicle, and as a profit seeking vehicle being caught must be part…
All three points in last paragraph seem wholy unrelated to each other and your larger point. I agreed with first half.
We all know it is super-easy to cheat in science, but not much can be done about it. Short of requiring replication for every result published which isn't feasible if the study was costly to produce. And the problem isn't confined to medicine either. How many studies in hpc of the type "we setup this benchmark and our novel algorithm/implementation won!" aren't also faked or flawed?
If the study is costly to produce, then it's even more important to replicate it, otherwise you'd wasting a large amount of money on a study with no real sense of whether it is flawed or not.
I've read that the real reason why researches generated by war crimes are worthless, is because we can't replicate it. Using a result that its validity can't be checked is just a way to disaster. And this is researches with the highest cost (human lives) and we still can't use it.
Lots of "interesting" psychology experiments in the days of yore turns out to have lots of damaging confounding variables and we can't just redo the experiment because, well, we shouldn't.
Pretty sure most studies about the effects of coffee and red wine are flawed.
why coffee and red wine specifically?
It might be that those are common enough in shared datasets or when datasets are collected. So it is easy to draw interferences with them and various other measured factors.