Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why isn't preprint review being adopted?

theroadgoeson.com

Post reply on HN
Print view View original

21–30 of 169 posts

Re: Why isn't preprint review being adopted?

#21

Post by bumby »

s0rce wrote: post #6
timr wrote: post #2

Even if academics could review all the papers on a preprint server (which the article argues -- rightly -- that they can't), it wouldn't solve the perceived problems (or the actual problems) with scientific peer review. The vast majority of irreproducible papers aren't detectible as irreproducible at time of publication. They look fine, and many actually are fine . They just don't reproduce. That's an expected outcom…

There is no place to make a note that something doesn't reproduce so its extremely time consuming or you need some source of tribal knowledge. In my postdoc I was trying to make some porous films and a bunch of paper's methods didn't seem to work, maybe I did it wrong, maybe some detail wasn't described, who knows. I couldn't get it to work and there was no way to document that failure.

I wonder if there's a way to document such replaceability failures as an erratum to the original manuscript. I feel like this would help in at least two major ways:

1) It provides a reproducibility filter. If a method isn't shown to be reproducible, publically documenting that adds to the body of knowledge, and this would help drive an incentive towards reproducing work rather than just searching for novelty. It would document work that would otherwise be lost because there's no incentive to showcase it. When the lack of reproducible results isn't public, it's now more likely that others may waste considerable effort in the same vein.

3) It may enlist the original authors to help understand why the work didn't reproduce well. Maybe the secondary effort lacked some crucial step or understanding. The people best positioned to remedy this are the original authors, and this secondary publication incentivizes them to dialogue with those who couldn't reproduce the outcome. It doesn't mean they have to engage, but it at least gives them some reason to involve themselves in the process.

Re: Why isn't preprint review being adopted?

#22

Post by nitwit005 »

rhelz wrote: post #10

One of my profs once remarked, "All of science is done on a volunteer basis." He was talking about peer review, which--as crucial as it is--is not something you get paid for. Writing a review--a good review--is 1) hard work, 2) can only be done by somebody who has spent years in postgraduate study, and 3) takes up a lot of time, which has many other demands on it. The solution? Its obvious. In a free market, how do y…

Unfortunately, what you'll actually incentivize is spending as little effort as possible to get the money paid out.

Re: Why isn't preprint review being adopted?

#23

Post by zer00eyz »

ajmurmann wrote: post #20
zer00eyz wrote: post #5

The review process is broken. Reviewing pre print papers isnt any more effective than reviewing printed papers. Review, and publication is a meaningless bar. Publish -> people find insight and try to pick it apart -> You either have flaws or you get reproduced... Only then should your paper be of any worth to be quoted or sighted from. The current system is glad-handing, intellectual protectionism and mastrubation. A…

"Publish -> people find insight and try to pick it apart -> You either have flaws or you get reproduced... Only then should your paper be of any worth to be quoted or sighted from." This is already how it's supposed to work. The review before publication is a fairly superficial check that just confirms that what you describe follows basic scientific practices. There is no validation of the actual research. A proper r…

There is a pretty simple way to change all of that.

Academic standards: You are not longer allowed to site a non reproduced paper in yours.

Citations matter as much as the print, put the hurdle there and all of a sudden things will change real quick.

Re: Why isn't preprint review being adopted?

#24

Post by matwood »

nitwit005 wrote: post #22
rhelz wrote: post #10

One of my profs once remarked, "All of science is done on a volunteer basis." He was talking about peer review, which--as crucial as it is--is not something you get paid for. Writing a review--a good review--is 1) hard work, 2) can only be done by somebody who has spent years in postgraduate study, and 3) takes up a lot of time, which has many other demands on it. The solution? Its obvious. In a free market, how do y…

Unfortunately, what you'll actually incentivize is spending as little effort as possible to get the money paid out.

Opposed to now where it appears lots of science is peer reviewed with all the problems found later?

Re: Why isn't preprint review being adopted?

#25

Post by matwood »

llm_trw wrote: post #3

Because I don't want to spend two years fighting the second referee when she's fundamentally misunderstood the point of my paper. I'm no longer in academia. Either take what I put up on arxiv or leave it. I _really_ don't care.

You actually touch on an interesting point. Is peer review still necessary? When there is limited journal space, sure. But now that we have effectively unlimited space, let it be reviewed when people go to cite it. We've already seen a lot of peer reviewed papers be retracted later, so should we just accept the reality?

Re: Why isn't preprint review being adopted?

#26

Post by nitwit005 »

matwood wrote: post #24
nitwit005 wrote: post #22

Earlier quoted context omitted.

Unfortunately, what you'll actually incentivize is spending as little effort as possible to get the money paid out.

Opposed to now where it appears lots of science is peer reviewed with all the problems found later?

Replacing a broken system, with a broken and also expensive system, does not sound like an improvement.

Re: Why isn't preprint review being adopted?

#28

Post by queuebert »

bumby wrote: post #16
s0rce wrote: post #7

Earlier quoted context omitted.

While not always true my metric for clear/understandable is for other people to understand it. This usually supports my argument when people show me a document and I have no idea what its saying, they argue its perfectly clear... my definition was for people other than the author to grasp the intended meaning.

If the intended audience is reading it, I would agree. However, many reviewers seem to be assigned and agree to review a topic that they are ill-equipped to understand without a substantial amount of background knowledge. A good reviewer in this situation will review the referenced papers to bolster their understanding. A bad reviewer will expect everything to be spelled out within the manuscript, and, unfortunately,…

Precisely. The continued hyperspecialization of science shrinks the potential pool of qualified reviewers to zero. I rarely get knowledgeable reviews these days. Instead of disputing my methods, they complain about the font or figure styles, or raise questions that weren't answered in the paper because they are common knowledge in the field.

Re: Why isn't preprint review being adopted?

#29

Post by andyferris »

matwood wrote: post #24
nitwit005 wrote: post #22

Earlier quoted context omitted.

Unfortunately, what you'll actually incentivize is spending as little effort as possible to get the money paid out.

Opposed to now where it appears lots of science is peer reviewed with all the problems found later?

It might cost $100k - $1m (or more) to repeat the work and run replications. The $5k - $10k mentioned earlier would be enough to allow time reading and thinking and checking some stuff on pen-and-paper.

Re: Why isn't preprint review being adopted?

#30

Post by ramblenode »

nitwit005 wrote: post #22
rhelz wrote: post #10

One of my profs once remarked, "All of science is done on a volunteer basis." He was talking about peer review, which--as crucial as it is--is not something you get paid for. Writing a review--a good review--is 1) hard work, 2) can only be done by somebody who has spent years in postgraduate study, and 3) takes up a lot of time, which has many other demands on it. The solution? Its obvious. In a free market, how do y…

Unfortunately, what you'll actually incentivize is spending as little effort as possible to get the money paid out.

Is that different from any other job?
Post reply on HN