Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why isn't preprint review being adopted?

theroadgoeson.com

Post reply on HN
Print view View original

61–70 of 169 posts

Re: Why isn't preprint review being adopted?

#61

Post by nextos »

adastra22 wrote: post #38

PREPUBLICATION REVIEW IS BAD! STOP TRYING TO REINTRODUCE IT. Sorry for the all caps. Publishing papers without “peer review” isn’t some radical new concept—it’s how all scientific fields operated prior to ca. 1970. That’s about when the pace of article writing outstripped available pages in journals and this system of pre-publication review was adopted and formalized. For the first 300 years of science you published…

You are right. It is often a problem that established competitors get to review articles that contradict their work and, unsurprisingly, try to sabotage them. Incentives are not well aligned.

A good mid-ground is something like the non-profit journal ELife, where articles are accepted by an editor and published before review, then reviewed publicly by selected reviewers.

Very transparant, and also leaves room for article updates. See the whole process here: https://elifesciences.org/about/peer-review.

Re: Why isn't preprint review being adopted?

#62

Post by heikkilevanto »

bumby wrote: post #14
rhelz wrote: post #10

One of my profs once remarked, "All of science is done on a volunteer basis." He was talking about peer review, which--as crucial as it is--is not something you get paid for. Writing a review--a good review--is 1) hard work, 2) can only be done by somebody who has spent years in postgraduate study, and 3) takes up a lot of time, which has many other demands on it. The solution? Its obvious. In a free market, how do y…

I don't disagree with the proposed idea of paying for review, but I would prefer also to have guardrails to ensure a good review. I would be willing to pay for a good review because it makes the paper/investigation better. But let's face it: under the current paradigm, there are also a lot of really bad reviews. It's one thing when it's apparent that a reviewer doesn't understand something because of a lack of clarit…

So, we just need a meta-review to review the reviews. At a cost, of course. And in order to keep that honest, we need a meta-meta-review...

Re: Why isn't preprint review being adopted?

#63

Post by rhelz »

geysersam wrote: post #40
rhelz wrote: post #10

One of my profs once remarked, "All of science is done on a volunteer basis." He was talking about peer review, which--as crucial as it is--is not something you get paid for. Writing a review--a good review--is 1) hard work, 2) can only be done by somebody who has spent years in postgraduate study, and 3) takes up a lot of time, which has many other demands on it. The solution? Its obvious. In a free market, how do y…

Or, research just gets published online free of charge for everyone to access, and important work will prove itself over time by being discovered, discussed and by becoming influential. If anyone wants to change something about an article (the writing, the structure of the paper, or anything else a reviewer might want to edit) they can just do it and publish a new version. If people like the new version better, good,…

Whether or not your papers pass peer review—-and which journals it is published in—are important criteria for hiring, tenure, whether your grants are funded, etc.

If you get rid of peer review, it’s not science. It’s just a vanity press.

Re: Why isn't preprint review being adopted?

#64

Post by ramblenode »

zer00eyz wrote: post #23
ajmurmann wrote: post #20

Earlier quoted context omitted.

"Publish -> people find insight and try to pick it apart -> You either have flaws or you get reproduced... Only then should your paper be of any worth to be quoted or sighted from." This is already how it's supposed to work. The review before publication is a fairly superficial check that just confirms that what you describe follows basic scientific practices. There is no validation of the actual research. A proper r…

There is a pretty simple way to change all of that. Academic standards: You are not longer allowed to site a non reproduced paper in yours. Citations matter as much as the print, put the hurdle there and all of a sudden things will change real quick.

> Academic standards: You are not longer allowed to site a non reproduced paper in yours.

> Citations matter as much as the print, put the hurdle there and all of a sudden things will change real quick.

The reproducibility crisis is just a symptom of the publish-or-perish culture that pushes academics to churning out bad research. Academia already over-emphasizes publishing positive results at the expense of studying important questions. Your solution would further incentivize low risk, low impact research that we have too much of.

Aside from that, there are a lot of edge cases that would make this difficult. If I do five studies that are modifications of each other, and all show the same basic effect, but I publish it as one paper, does that count as being reproduced? What if a result has only ever been reproduced within a single research group? Does the Higgs Boson need to be confirmed at at a collider outside the LHC?

Re: Why isn't preprint review being adopted?

#65

Post by YeGoblynQueenne »

pacbard wrote: post #9
timr wrote: post #2

Even if academics could review all the papers on a preprint server (which the article argues -- rightly -- that they can't), it wouldn't solve the perceived problems (or the actual problems) with scientific peer review. The vast majority of irreproducible papers aren't detectible as irreproducible at time of publication. They look fine, and many actually are fine . They just don't reproduce. That's an expected outcom…

To your second point, I alway go back to this quote: "You can't fix by analysis what you bungled by design" (Light, Singer and Willett, 1990). If a paper is broken by design, there isn't much to do after the fact. It's just broken. The problem is that doing a good RCT takes both time and effort, with the huge risk of having null results, which usually results in a desk rejection from most top journals. So, you either…

>> (Light, Singer and Willett, 1990)

A citation like the one above should normally point to a full reference in a bibliography section. Did you forget the \bibliography{} command at the end of your comment?

Re: Why isn't preprint review being adopted?

#66

Post by adastra22 »

nextos wrote: post #61
adastra22 wrote: post #38

PREPUBLICATION REVIEW IS BAD! STOP TRYING TO REINTRODUCE IT. Sorry for the all caps. Publishing papers without “peer review” isn’t some radical new concept—it’s how all scientific fields operated prior to ca. 1970. That’s about when the pace of article writing outstripped available pages in journals and this system of pre-publication review was adopted and formalized. For the first 300 years of science you published…

You are right. It is often a problem that established competitors get to review articles that contradict their work and, unsurprisingly, try to sabotage them. Incentives are not well aligned. A good mid-ground is something like the non-profit journal ELife, where articles are accepted by an editor and published before review, then reviewed publicly by selected reviewers. Very transparant, and also leaves room for art…

That’s a better system. But why involve the journal in review at all?

Journals should go back to just publishing papers and any unsolicited letter-to-the-editor reviews, reproductions, or commentary they receive in response. Why add a burden of unpaid work reviewing every single paper that comes through?

Re: Why isn't preprint review being adopted?

#67

Post by ska »

geysersam wrote: post #40
rhelz wrote: post #10

One of my profs once remarked, "All of science is done on a volunteer basis." He was talking about peer review, which--as crucial as it is--is not something you get paid for. Writing a review--a good review--is 1) hard work, 2) can only be done by somebody who has spent years in postgraduate study, and 3) takes up a lot of time, which has many other demands on it. The solution? Its obvious. In a free market, how do y…

Or, research just gets published online free of charge for everyone to access, and important work will prove itself over time by being discovered, discussed and by becoming influential. If anyone wants to change something about an article (the writing, the structure of the paper, or anything else a reviewer might want to edit) they can just do it and publish a new version. If people like the new version better, good,…

> Just make it accessible.

We already have that system, it's called the internet. Nothing stops you or I from putting our ideas online for all to read, comment on, update, etc.

The role of the publishers, flawed as it is, has little to do with the physical cost of producing or providing an article, and is filling (one can argue badly) a role in curation and archival that is clearly needed. Any proposal to change the system really has to address how those roles are met, hopefully cheaper than currently but definitely not more expensive because mostly people don't get paid (in $ or career or anything) now - or has to provide a funding source for it.

I don't really see how your outlined scenario addresses that, at least not in a way that's functionally different than today. Can you expand?

Re: Why isn't preprint review being adopted?

#68

Post by notatoad »

heikkilevanto wrote: post #62
bumby wrote: post #14

Earlier quoted context omitted.

I don't disagree with the proposed idea of paying for review, but I would prefer also to have guardrails to ensure a good review. I would be willing to pay for a good review because it makes the paper/investigation better. But let's face it: under the current paradigm, there are also a lot of really bad reviews. It's one thing when it's apparent that a reviewer doesn't understand something because of a lack of clarit…

So, we just need a meta-review to review the reviews. At a cost, of course. And in order to keep that honest, we need a meta-meta-review...

we need twitter community notes for science

Re: Why isn't preprint review being adopted?

#69

Post by nextos »

adastra22 wrote: post #66
nextos wrote: post #61

Earlier quoted context omitted.

You are right. It is often a problem that established competitors get to review articles that contradict their work and, unsurprisingly, try to sabotage them. Incentives are not well aligned. A good mid-ground is something like the non-profit journal ELife, where articles are accepted by an editor and published before review, then reviewed publicly by selected reviewers. Very transparant, and also leaves room for art…

That’s a better system. But why involve the journal in review at all? Journals should go back to just publishing papers and any unsolicited letter-to-the-editor reviews, reproductions, or commentary they receive in response. Why add a burden of unpaid work reviewing every single paper that comes through?

I believe ELife may eventually get deeper paid reviews. That is a reason to involve the journal in this process. The way reviews at ELife work can be seen as solicited letter-to-the editor reviews, as these do not influence the outcome. Your article is already published.

Re: Why isn't preprint review being adopted?

#70

Post by coolhand2120 »

adastra22 wrote: post #38

PREPUBLICATION REVIEW IS BAD! STOP TRYING TO REINTRODUCE IT. Sorry for the all caps. Publishing papers without “peer review” isn’t some radical new concept—it’s how all scientific fields operated prior to ca. 1970. That’s about when the pace of article writing outstripped available pages in journals and this system of pre-publication review was adopted and formalized. For the first 300 years of science you published…

https://en.wikipedia.org/wiki/Replication_crisis

The reputation of scientific researchers has been greatly harmed by the current system. Please, help find a way to fix it, or at the very least don't hinder people trying to fix it. Thanks to the way we do things now a coin flip is _better_ than peer review. Public trust in science is at an all time low. I really hope you don't think "this is fine".

Post reply on HN