Why isn't preprint review being adopted?
theroadgoeson.com
Why isn't preprint review being adopted?
1–10 of 169 posts
Re: Why isn't preprint review being adopted?
#2Even if academics could review all the papers on a preprint server (which the article argues -- rightly -- that they can't), it wouldn't solve the perceived problems (or the actual problems) with scientific peer review.
The vast majority of irreproducible papers aren't detectible as irreproducible at time of publication. They look fine, and many actually are fine. They just don't reproduce. That's an expected outcome in science. The system will self-correct over time.
IMO, the main actual problem with peer review is that non-practitioners put too much faith in it. Nobody in science actually takes a paper on faith because it's been published, and you shouldn't either. Peer review is little more than a lightweight safeguard against complete nonsense being published. It barely works for that. Just because you found a paper doesn't mean you should believe it. You have to understand it.
A secondary actual problem is that it's impossible to reproduce a lot of papers, or they're methodologically broken from the start (e.g. RCTs that are not pre-registered, or observational studies without control groups). These are problems we could actually solve. For example, just requiring that any paper publish the raw study data would help to self-control the system. There are high-profile researchers out there, right now, who do little more than statistically mine the same secret data set -- these people are likely publishing crap, but we have no way to prove it, because the data is secret.
Re: Why isn't preprint review being adopted?
#3Because I don't want to spend two years fighting the second referee when she's fundamentally misunderstood the point of my paper.
I'm no longer in academia. Either take what I put up on arxiv or leave it. I _really_ don't care.
Re: Why isn't preprint review being adopted?
#4I would worry that preprint review would turn into another front of the culture wars for certain fields and science by consensus.
Re: Why isn't preprint review being adopted?
#5The review process is broken.
Reviewing pre print papers isnt any more effective than reviewing printed papers. Review, and publication is a meaningless bar.
Publish -> people find insight and try to pick it apart -> You either have flaws or you get reproduced... Only then should your paper be of any worth to be quoted or sighted from.
The current system is glad-handing, intellectual protectionism and mastrubation.
Academia has only itself to blame for this, and they are apparently unwilling to fix it.
Re: Why isn't preprint review being adopted?
#6Even if academics could review all the papers on a preprint server (which the article argues -- rightly -- that they can't), it wouldn't solve the perceived problems (or the actual problems) with scientific peer review. The vast majority of irreproducible papers aren't detectible as irreproducible at time of publication. They look fine, and many actually are fine . They just don't reproduce. That's an expected outcom…
There is no place to make a note that something doesn't reproduce so its extremely time consuming or you need some source of tribal knowledge. In my postdoc I was trying to make some porous films and a bunch of paper's methods didn't seem to work, maybe I did it wrong, maybe some detail wasn't described, who knows. I couldn't get it to work and there was no way to document that failure.
Re: Why isn't preprint review being adopted?
#7Because I don't want to spend two years fighting the second referee when she's fundamentally misunderstood the point of my paper. I'm no longer in academia. Either take what I put up on arxiv or leave it. I _really_ don't care.
While not always true my metric for clear/understandable is for other people to understand it. This usually supports my argument when people show me a document and I have no idea what its saying, they argue its perfectly clear... my definition was for people other than the author to grasp the intended meaning.
Re: Why isn't preprint review being adopted?
#8Even if academics could review all the papers on a preprint server (which the article argues -- rightly -- that they can't), it wouldn't solve the perceived problems (or the actual problems) with scientific peer review. The vast majority of irreproducible papers aren't detectible as irreproducible at time of publication. They look fine, and many actually are fine . They just don't reproduce. That's an expected outcom…
Peer review is a spam filter, and it’s quite useful in that content. But it’s a spam filer for people who filter out 99.9+% of papers by having such a narrow scope they probably recognize several names on a given paper.
Re: Why isn't preprint review being adopted?
#9Even if academics could review all the papers on a preprint server (which the article argues -- rightly -- that they can't), it wouldn't solve the perceived problems (or the actual problems) with scientific peer review. The vast majority of irreproducible papers aren't detectible as irreproducible at time of publication. They look fine, and many actually are fine . They just don't reproduce. That's an expected outcom…
To your second point, I alway go back to this quote: "You can't fix by analysis what you bungled by design" (Light, Singer and Willett, 1990).
If a paper is broken by design, there isn't much to do after the fact. It's just broken.
The problem is that doing a good RCT takes both time and effort, with the huge risk of having null results, which usually results in a desk rejection from most top journals.
So, you either are a top-fund raising researcher who can both fund multiple RCTs and people to support them, or you just try your best with what you have and hope to squeeze a paper out from you did.
Releasing the data won't really help much if the data generating process is flawed. Sure, other people will be able to run different kind of analyses (e.g., jackknife your standard errors instead of just using a robust correction), but I'm not sure how helpful that will be.
A third issue that I have also encountered is that journal editors have an agenda when putting together an issue, which sometimes overwrites the "quality' of the research with "fit" to the issue. This could lead to "lower quality" articles to be published because they fit the (often unspoken) direction of the journal. Most editors see their role as steering the field towards new directions (a sort of a meta service to the field) and sometimes that comes at the expense of the quality of the work.
Re: Why isn't preprint review being adopted?
#10One of my profs once remarked, "All of science is done on a volunteer basis." He was talking about peer review, which--as crucial as it is--is not something you get paid for.
Writing a review--a good review--is 1) hard work, 2) can only be done by somebody who has spent years in postgraduate study, and 3) takes up a lot of time, which has many other demands on it.
The solution? Its obvious. In a free market, how do you signal if you want more of something to be produced? Bueller? Bueller?
Yeah, that's right, you gotta pay for it. This cost should just be estimated and factored into the original grant proposals--if its not worth $5k or $10k to fund a round of peer review, and perhaps also funds to run confirming experiments--well, then its probably not research worth doing in the first place.
So yeah, write up the grants to include the actual full cost of doing and publishing the research. It would be a great way for starving grad students to earn some coin, and the experience gained in running confirming experiments would be invaluable to help them get that R.A. position or postdoc.