Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why isn't preprint review being adopted?

theroadgoeson.com

Post reply on HN
Print view View original

121–130 of 169 posts

Re: Why isn't preprint review being adopted?

#121

Post by matthewdgreen »

rhelz wrote: post #104
matthewdgreen wrote: post #94

Earlier quoted context omitted.

If you keep increasing publication requirements in a Red Queen’s race, you both increase the amount of peer-review required and also reduce the time that qualified scientists have to do it. No matter how much people want to build a functioning system, they only have so many hours in the day.

A valid concern, but what is driving this red-queen race is that you are competing with people who cheat. If you’ve got one really solid paper published, but your competitor fraudulently published 10, it’s real tempting to fraud a few papers yourself. But they multiply fraudulent papers because they can get away with it, and they can get away with it because nobody is really reviewing or replicating those results. I…

The idea that fraud is rampant and driving the explosion in publication rates is deeply tempting to folks who don’t work directly in a scientific field. But it’s a misconception that’s largely driven by non-scientific media and bloggers. In practice literal fraud does exist, but it’s relatively rare in most scientific fields. The explosion in publication rates is largely caused by a combination of (1) more people entering the field, and (2) “paper slicing” where people submit smaller contributions as part of each paper, in order to maximize publication metrics.

As far as “no publication without replicating the results”. Think for a second about how that will work. How is anyone going to replicate a result if it’s not published? This is literally the reason that scientific publication exists in the first place: to provide a channel by which new results can be conveyed to other scientists so that they can learn from, and in some cases, replicate those results. So clearly you don’t want to prevent publication of non-replicated results, you just want to prevent people from getting some kind of academic brownie-point credit for those publications. Perhaps that’s workable, I don’t know. But you need to be a lot more knowledgeable and curious about the scientific publication system in order to have any hope of improving it. Taken literally, your “no publication without replication” proposal would inhibit scientific replication entirely.

Re: Why isn't preprint review being adopted?

#122

Post by dbingham »

adastra22 wrote: post #113
dbingham wrote: post #72

Earlier quoted context omitted.

Hi! Author here. Preprint review as it is being discussed here is post-publication. The preprint is shared first, and review layered on top of it later. Click through to the paper[1] I'm responding to and give it a read. But, also, prepublication review doesn't need to be "reintroduced". It's still the standard for the vast majority of scholarship. By some estimates there are around 5 million scholarly papers publish…

Thanks for your charitable response to a reader who quite obviously misunderstood what you were writing about. You might want to update your article to make that clear, as "preprint review" isn't a term I had ever encountered before, and a simple reading of your article doesn't obviously indicate that you were talking about post-publication review. (Note that outside of academia, upload to arXiv is publication, so ye…

No worries. I was really writing for an audience of academics and, in particular, people involved in the various open science and publication reform movements. Sharing here was a bit of an after thought, so it wasn't written for an audience unfamiliar with the intricacies of academic publishing. It's a complicated enough space as it is that this is already a 4-5 article series with the potential to grow, even when I'm writing with the assumption of quite of a bit of knowledge.

But yeah, "preprint review" is considered post-publication review both in and outside of academia. There are nuances to what is consider "publication" in academia. A preprint is not a "Version of Record", meaning it doesn't count towards tenure and promotion. The movement pushing for preprint review is attempting to layer review on top of already public preprints in the hopes that reviewed preprints can begin to count as VORs. It's unclear whether that will work.

Some models, like eLife's, seem more promising than others. But eLife got a ton of backlash when they switched to their new reviewed preprint model, so it remains to be seen whether it will work in the long run.

Re: Why isn't preprint review being adopted?

#123

Post by dbingham »

MichaelZuo wrote: post #119
dbingham wrote: post #72

Earlier quoted context omitted.

Hi! Author here. Preprint review as it is being discussed here is post-publication. The preprint is shared first, and review layered on top of it later. Click through to the paper[1] I'm responding to and give it a read. But, also, prepublication review doesn't need to be "reintroduced". It's still the standard for the vast majority of scholarship. By some estimates there are around 5 million scholarly papers publish…

Quality control was handled fine enough by editors for literally all output on the planet pre 1970s. There’s nothing physically stopping that paradigm from returning now that we have the internet, other than the fact that there’s only a few thousand (?) such editors. If somehow there were fifty thousand such editors, then the whole peer review system would be completely unnecessary. Of course not enough people want t…

There are nearly 50,000 commercial journals and a long tail of non-commercial journals each with teams of editors. There are probably hundreds of thousands of people currently serving as editors.

The issue isn't editorial bandwidth, it's that peer review is currently built into the promotion and tenure structure for academics, who produce the vast majority of scholarship and thus dictate the shape of the scholarly publishing system.

Academics have to publish the papers in peer reviewed journals for it to count towards tenure and promotion. And in fact, they are limited to a small set of journals that are deemed high quality enough for their fields. These journals are chosen by tenure and promotion committees composed of their senior peers and school administration. There are over 1000 R1 and R2 universities worldwide, each with hundreds of departments each with their own tenure and promotion committees. So changing the system is a massive collective action problem.

Re: Why isn't preprint review being adopted?

#124

Post by staunton »

ska wrote: post #97
paulmd wrote: post #96

Earlier quoted context omitted.

> We already have that system, it's called the internet. Nothing stops you or I from putting our ideas online for all to read, comment on, update, etc. Are you asking how arxiv is different from blogspot?

No, although the mechanism for hosting the content aren’t that important. Preprint servers are very useful but haven’t replaced journals for good reasons.

What are those reasons? The only thing I see that journals do which preprint servers couldn't easily take over is prestige.

Re: Why isn't preprint review being adopted?

#125

Post by PaulHoule »

rhelz wrote: post #10

One of my profs once remarked, "All of science is done on a volunteer basis." He was talking about peer review, which--as crucial as it is--is not something you get paid for. Writing a review--a good review--is 1) hard work, 2) can only be done by somebody who has spent years in postgraduate study, and 3) takes up a lot of time, which has many other demands on it. The solution? Its obvious. In a free market, how do y…

And also has limited value as we practice it today.

A useful review would involve:

(a) "This paper won't be accepted by Cochrane for meta analysis", "N=20 get out of here", ...

(b) Researchers provided their data files and Jupyter notebooks, the reviewer got them to run

(c) Reviewers attempt their own analysis for at least some of the data (think of the model of accounting where auditors look at a sample of the books)

(d) Reviewers come visit the lab and take a look at the apparatus

(e) Something like a PhD defense

(f) Summarize/formalize discussion of a paper that happens at a conference or online venue

(g) 5 months, 5 years, or 40 years later (once in my case) somebody goes through the math line-by-line and finds a mistake on equation 178 that propagates and affects the rest of the calculation. This knowledge ought to be captured and "stapled" to the paper.

I wouldn't say peer review is useless, I think it did improve papers I wrote a little but reviewers do not put enough effort into to reliably catch validity problems. If you believe in meta-analysis, which you should, read this book

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1114087/

the first one (a) is really important because the overwhelming majority of papers are excluded in Cochrane reviews for most medical topics. If 80-90% of papers in some fields are not suitable for inference that raises tough questions such as "Why does work like this get published?" and "Why does this work get funded?" If we got half as many papers but 80-90% of them were usable that would be a revolution.

Re: Why isn't preprint review being adopted?

#126

Post by roflmaostc »

somethingsome wrote: post #78

A fun system would be: you 'have to' peer review publicly a (part of a) paper (at least) when you cite it in your paper. So that often cited paper get a lot of different (small) public reviews that can be curated from time to time, obscure papers get at least one review justifying why it's relevant to cite them in the new work. Some could argue that this is too much work added to the writing process.. But.. At the sa…

Yeah, such a system would be great.

A bit like Google Scholar. Papers are indexed and you can access the references easily. And you would be able to comment and review certain lines. Everyone could add notes that certain equations are wrong, etc. In best case authors would engage in the discussion too.

But obviously this won't work because some papers are behind paywalls :/

Re: Why isn't preprint review being adopted?

#127

Post by bumby »

rhelz wrote: post #105
bumby wrote: post #102

Earlier quoted context omitted.

Then don’t you think the first part of that process is explaining the methods and results of your experiment? That’s precisely what the current situation does. What’s lacking is the replication incentives.

I propose to supply those incentives—-pay for good reviewer and replicators. If you want something done, you gotta pay for it. We can’t just rely on volunteers.

I'm on board with that idea, as long as we can also provide guardrails against the perverse incentives of paying for them. E.g., we need to avoid frivolous reviews/replication as well as something evolving into a "pay for a good review" service.

Re: Why isn't preprint review being adopted?

#130

Post by abhgh »

adastra22 wrote: post #115
abhgh wrote: post #109

Earlier quoted context omitted.

Agree. With more compute-intensive experiments in ML, I am not sure if any reviewer re-runs experiments. The reviewing is already voluntary - who's going to pay out of their pocket for the compute? It's not feasible.

Re-running experiments is out of scope for any reviewer. You're supposed to trust their data, but review that their conclusion is actually supported by the data, as shown.

Yeah, I agree. But I said what I did to make the larger point that empirical studies are hard to vouch for without some level of trust on both sides. For ex if someone selects favorable datasets, metrics, "random" seeds, p-hacking etc.
Post reply on HN