Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why isn't preprint review being adopted?

theroadgoeson.com

Post reply on HN
Print view View original

71–80 of 169 posts

Re: Why isn't preprint review being adopted?

#71

Post by currymj »

heikkilevanto wrote: post #62
bumby wrote: post #14

Earlier quoted context omitted.

I don't disagree with the proposed idea of paying for review, but I would prefer also to have guardrails to ensure a good review. I would be willing to pay for a good review because it makes the paper/investigation better. But let's face it: under the current paradigm, there are also a lot of really bad reviews. It's one thing when it's apparent that a reviewer doesn't understand something because of a lack of clarit…

So, we just need a meta-review to review the reviews. At a cost, of course. And in order to keep that honest, we need a meta-meta-review...

many CS conferences have something literally called a "meta-review" and then there are further senior people who read and oversee the meta-reviews. it stops there though.

Re: Why isn't preprint review being adopted?

#72

Post by dbingham »

adastra22 wrote: post #38

PREPUBLICATION REVIEW IS BAD! STOP TRYING TO REINTRODUCE IT. Sorry for the all caps. Publishing papers without “peer review” isn’t some radical new concept—it’s how all scientific fields operated prior to ca. 1970. That’s about when the pace of article writing outstripped available pages in journals and this system of pre-publication review was adopted and formalized. For the first 300 years of science you published…

Hi! Author here.

Preprint review as it is being discussed here is post-publication. The preprint is shared first, and review layered on top of it later. Click through to the paper[1] I'm responding to and give it a read.

But, also, prepublication review doesn't need to be "reintroduced". It's still the standard for the vast majority of scholarship. By some estimates there are around 5 million scholarly papers published per year. There are only about 10 - 20 million preprints published total over the past 30 years since Arxiv's introduction.

There are a bunch of layered institutions and structures that are maintaining it as the standard. I don't have data for it to hand, but my understanding is that the vast majority of preprints go on to be published in a journal with pre-publication review. And as far as most of the institutions are concerned, papers aren't considered valid until they have published in a journal with prepublication review.

There is a significant movement pushing for the adoption of preprint review as an alternative to journal publishing with the hope that it can begin a change to this situation.

The idea is that preprint review offers a similar level of quality control as journal review (which, most reformers would agree is not much) and could theoretically replace it in those institutional structures. That would, at least invert the current process: with papers being shared immediately and review coming later after the results were shared openly.

[1] https://journals.plos.org/plosbiology/article?id=10.1371/jou...

Re: Why isn't preprint review being adopted?

#73

Post by timr »

ramblenode wrote: post #49
timr wrote: post #2

Even if academics could review all the papers on a preprint server (which the article argues -- rightly -- that they can't), it wouldn't solve the perceived problems (or the actual problems) with scientific peer review. The vast majority of irreproducible papers aren't detectible as irreproducible at time of publication. They look fine, and many actually are fine . They just don't reproduce. That's an expected outcom…

> The vast majority of irreproducible papers aren't detectible as irreproducible at time of publication. They look fine, and many actually are fine. They just don't reproduce. That's an expected outcome in science. This is not entirely true. A power analysis is how you determine reproducibility, and researchers should be doing it before they begin collecting data. Reviewers can do it post-hoc with assumptions about t…

> A power analysis is how you determine reproducibility

All a power analysis does is reduce the chance that the result is a false negative. It doesn't reduce the chance of a false positive.

> False positives produce inflated effect sizes

Not always. Lots of studies publish as "significant" as soon as they get a p-value just under .05. Inflated effect sizes are certainly a sign that something could be wrong, but it's just one indicator.

Regardless, even if you have a power analysis at the conventional threshold of 80%, and a p-value of .05, you're still going to get spurious positive results 5% of the time, and spurious negative results 20% of the time, by definition.

Re: Why isn't preprint review being adopted?

#74

Post by somethingsome »

coolhand2120 wrote: post #70
adastra22 wrote: post #38

PREPUBLICATION REVIEW IS BAD! STOP TRYING TO REINTRODUCE IT. Sorry for the all caps. Publishing papers without “peer review” isn’t some radical new concept—it’s how all scientific fields operated prior to ca. 1970. That’s about when the pace of article writing outstripped available pages in journals and this system of pre-publication review was adopted and formalized. For the first 300 years of science you published…

https://en.wikipedia.org/wiki/Replication_crisis The reputation of scientific researchers has been greatly harmed by the current system. Please, help find a way to fix it, or at the very least don't hinder people trying to fix it. Thanks to the way we do things now a coin flip is _better_ than peer review. Public trust in science is at an all time low. I really hope you don't think "this is fine".

IMO 'public trust in science is at an all time low' is because of bad journalism more than bad papers.

Papers are written by academy-type individuals for academy-type individuals, not for consumption by non experts. An academic is usually pretty fast to determine if a paper is to be trusted.

So interpreting and extrapolating to the extreme the results of a minor paper in an obscure journal is more bad journalism than bad science.

Then we wonder why people don't trust science..

Re: Why isn't preprint review being adopted?

#75

Post by zer00eyz »

ramblenode wrote: post #64
zer00eyz wrote: post #23

Earlier quoted context omitted.

There is a pretty simple way to change all of that. Academic standards: You are not longer allowed to site a non reproduced paper in yours. Citations matter as much as the print, put the hurdle there and all of a sudden things will change real quick.

> Academic standards: You are not longer allowed to site a non reproduced paper in yours. > Citations matter as much as the print, put the hurdle there and all of a sudden things will change real quick. The reproducibility crisis is just a symptom of the publish-or-perish culture that pushes academics to churning out bad research. Academia already over-emphasizes publishing positive results at the expense of studying…

>> What if a result has only ever been reproduced within a single research group? Does the Higgs Boson need to be confirmed at at a collider outside the LHC?

Yes. It does. Take out all the detectors and let another team build their own and come in and prove it. The really expensive part was the big concrete doughnut in the ground. After that wee could ... you know... shut the fuckin thing down so we stop pouring money into a literal hole in the ground. Idk we could do some more science after that. What is missing from quantum theory that the LHC running is going to find?

IF we stopped funding mastrubatory string theory and put more of that into practical physics maybe we would have had something else for the LHC to do...

>>> Academia already over-emphasizes publishing positive results at the expense of studying important questions

Academia is made up of Academics, they aren't inclined to fix their own problem... it as effective as the church policing its own issues.

I love what Jack Horner has done (see: https://creation.com/dino-puberty-blues ) but he would never have been able to achieve any of that if those people were alive and his peers.

Re: Why isn't preprint review being adopted?

#76

Post by 1letterunixname »

Up until the 1940's, "publish or perish" wasn't the obsession, publishing what was thoroughly vetted but not before it was ready. The sheer volume of substandard, barely novel papers allowed and the artificial expectations to produce a blizzard of publications foisted on researchers are the central problems.

Re: Why isn't preprint review being adopted?

#77

Post by zer00eyz »

freedomben wrote: post #48
zer00eyz wrote: post #23

Earlier quoted context omitted.

There is a pretty simple way to change all of that. Academic standards: You are not longer allowed to site a non reproduced paper in yours. Citations matter as much as the print, put the hurdle there and all of a sudden things will change real quick.

> You are not longer allowed to site a non reproduced paper in yours. I fully agree that in an ideal world, that would be the case. But some reproductions (especially now with machine learning) could cost millions of dollars and years to do. I don't think that's a reasonable or feasible thing to require.

>> could cost millions of dollars and years to do

https://en.wikipedia.org/wiki/List_of_colleges_and_universit...

Not only do they have the money they are charging the students more than ever.

Re: Why isn't preprint review being adopted?

#78

Post by somethingsome »

A fun system would be: you 'have to' peer review publicly a (part of a) paper (at least) when you cite it in your paper.

So that often cited paper get a lot of different (small) public reviews that can be curated from time to time, obscure papers get at least one review justifying why it's relevant to cite them in the new work.

Some could argue that this is too much work added to the writing process.. But.. At the same time.. Shouldn't we read the papers we cite? Why not automatically write a small review of it? It has not to be huge, only the justification on why we (can) use it in our work.

Re: Why isn't preprint review being adopted?

#79

Post by warkdarrior »

bumby wrote: post #21
s0rce wrote: post #6

Earlier quoted context omitted.

There is no place to make a note that something doesn't reproduce so its extremely time consuming or you need some source of tribal knowledge. In my postdoc I was trying to make some porous films and a bunch of paper's methods didn't seem to work, maybe I did it wrong, maybe some detail wasn't described, who knows. I couldn't get it to work and there was no way to document that failure.

I wonder if there's a way to document such replaceability failures as an erratum to the original manuscript. I feel like this would help in at least two major ways: 1) It provides a reproducibility filter. If a method isn't shown to be reproducible, publically documenting that adds to the body of knowledge, and this would help drive an incentive towards reproducing work rather than just searching for novelty. It woul…

2) ???

Re: Why isn't preprint review being adopted?

#80

Post by bumby »

freedomben wrote: post #43
adastra22 wrote: post #38

PREPUBLICATION REVIEW IS BAD! STOP TRYING TO REINTRODUCE IT. Sorry for the all caps. Publishing papers without “peer review” isn’t some radical new concept—it’s how all scientific fields operated prior to ca. 1970. That’s about when the pace of article writing outstripped available pages in journals and this system of pre-publication review was adopted and formalized. For the first 300 years of science you published…

> The idea of having your intellectual competitors review your work potentially sabotage your publication prospects as a standard process is a relatively recent addition. And one that has not been shown to actually be effective. If this is true (and I'm not doubting you, just acknowledging that I'm taking your word for it) then why abandon the entire system? Why not just roll it back to the state before we added the…

>Why not just roll it back to the state before we added the intellectual competitor review?

The only thing I can currently think of is that the pace of research* has grown so much that a small group of editors may be unable to handle the amount of submissions. This could result in a) an inability of the editors to thoroughly vet the submissions, b) difficulty in "good" submissions being found (ie, separating the wheat from the chafe), or c) a further devolution into very, very niche journals just to make the scope manageable for the editors

* I would concede that a very, very large proportion of current research is either heavily derivative or auto-cited, so overall growth isn't to be conflated with growth in quality research

Post reply on HN