Cracking down on research fraud
undark.org
Cracking down on research fraud
1–10 of 227 posts
Re: Cracking down on research fraud
#2[deleted]
Re: Cracking down on research fraud
#3I think one of the big things that you can do is split up the data gathering and data analysis.
Kind of like we don’t trust the companies to audit themselves, instead we have an outside firm.
In this model, a researcher would create a hypothesis and collect the data. Their team would write the background and methods sections of the paper.
Then the entirety of raw data would be sent off to a third party for data analysis and they would write the results part of the paper.
The original team would then write the discussion part discussing the implications of the study.
All papers would be required to be made public.
The idea is that there would be specialized firms that do the analysis on the raw data for everyone. These would be carefully audited and certified by the government. They have no incentive to play statistical games. If they get caught cheating, then they have to pay for all the analysis on all the papers to be done again by a competitor and if any errors are found in a paper analysis it is automatically retracted. While the this re-analysis is going on, all papers would be quarantined with a note stating the paper is having its analysis redone. This incentivizes the analysis firm to be ethical, as well as incentivizes researchers to pick ethical analysis firms.
Separating data collection from data analysis would help align incentives better.
Re: Cracking down on research fraud
#4Basic set of checks-and-balances:
* Preregistration and adoption of open science practices
* Public access to research results, methods, and data, with some exceptions (such as PII)
* Federally-funded universities can't use NDAs or non-disparage agreements
* Federally-funded universities must respond to records requests under terms similar to FOIA (note that FOIA has requestor pay costs)
* Federally-funded universities must adopt transparent governance
* Salary caps at federally-funded universities and affiliated organizations
* Conflict-of-interest laws with hard enforcement
* Federally-funded universities must publicly publish research misconduct and alleged research misconduct. The latter is tricky, since you don't want to smear the researcher without proof, but you also don't want to trust results.
This really needs reform.
Re: Cracking down on research fraud
#5I think one of the big things that you can do is split up the data gathering and data analysis. Kind of like we don’t trust the companies to audit themselves, instead we have an outside firm. In this model, a researcher would create a hypothesis and collect the data. Their team would write the background and methods sections of the paper. Then the entirety of raw data would be sent off to a third party for data analy…
That's an interesting idea that could stop very specific types of fraud, certainly in the life sciences. But it's not feasible for all kinds of research, and in fact could hinder lots of research.
> All papers would be required to be made public.
This is more universally feasible. Publicizing the data and analysis tools (scripts, software) falls into the same category, and would go a long way to help without the need for such strong separation.
Re: Cracking down on research fraud
#6The root of the problem is that we allow people with high esteemed credentials to run society. They jumped through a hoop.
"You got straight As at 14 - 18 years old and got into an ivy league school as a result? Here run this venture fund."
"You got a PhD in Economics with a good publication by P-hacking your secret data? Here take a run at the FED with power over the US Economy. Your Phd shows that you are the man for the job."
"You got a PhD in ML by making some incremental improvement on some already existing model and then doing massive hyper parameter tuning? Here, become director of research at this big corporation."
Research will never be fully productive in this system, there are too many people who have too much to gain from gaming the publication system.
Re: Cracking down on research fraud
#7Basic set of checks-and-balances: * Preregistration and adoption of open science practices * Public access to research results, methods, and data, with some exceptions (such as PII) * Federally-funded universities can't use NDAs or non-disparage agreements * Federally-funded universities must respond to records requests under terms similar to FOIA (note that FOIA has requestor pay costs) * Federally-funded universiti…
Salary caps? So the best people go into the private sector where they can actually get paid?
Re: Cracking down on research fraud
#8The root of the problem is that we allow people with high esteemed credentials to run society. They jumped through a hoop. "You got straight As at 14 - 18 years old and got into an ivy league school as a result? Here run this venture fund." "You got a PhD in Economics with a good publication by P-hacking your secret data? Here take a run at the FED with power over the US Economy. Your Phd shows that you are the man f…
Thanks for pointing out the problem with credentials.
Name a better indicator then for being competent.
Re: Cracking down on research fraud
#9Basic set of checks-and-balances: * Preregistration and adoption of open science practices * Public access to research results, methods, and data, with some exceptions (such as PII) * Federally-funded universities can't use NDAs or non-disparage agreements * Federally-funded universities must respond to records requests under terms similar to FOIA (note that FOIA has requestor pay costs) * Federally-funded universiti…
Salary caps? So the best people go into the private sector where they can actually get paid?
Agree, I don't think salary caps have much to do with the poor alignment of incentives under discussion here.
Re: Cracking down on research fraud
#10The root of the problem is that we allow people with high esteemed credentials to run society. They jumped through a hoop. "You got straight As at 14 - 18 years old and got into an ivy league school as a result? Here run this venture fund." "You got a PhD in Economics with a good publication by P-hacking your secret data? Here take a run at the FED with power over the US Economy. Your Phd shows that you are the man f…
Thanks for pointing out the problem with credentials. Name a better indicator then for being competent.
Industry kinda already has. Products launched. Teams managed. A candidate's publication record is a conversation starter during interviews, but I feel like years of experience at a company working on launching a successful project is a much more valuable currency.