Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why isn't preprint review being adopted?

theroadgoeson.com

Post reply on HN
Print view View original

91–100 of 169 posts

Re: Why isn't preprint review being adopted?

#91

Post by kelipso »

SkyPuncher wrote: post #57
adastra22 wrote: post #38

PREPUBLICATION REVIEW IS BAD! STOP TRYING TO REINTRODUCE IT. Sorry for the all caps. Publishing papers without “peer review” isn’t some radical new concept—it’s how all scientific fields operated prior to ca. 1970. That’s about when the pace of article writing outstripped available pages in journals and this system of pre-publication review was adopted and formalized. For the first 300 years of science you published…

I've found the entire peer review concept in academia to be extremely odd. I'm not entirely sure what problem it solves. It seems like you have one of two types of people reading these articles: * People who are already specialists/familiar enough with concepts. They'll either call BS upfront or run experimentation themselves to validate results. * People who aren't specialists and will need to corroborate evidence a…

> run experimentation themselves to validate results

Let's just breeze through an that like it's nothing haha.

Also the people reading these peer reviewed articles range from new grad students to researchers with decades of experience.

There are many ways to see whether an article is high quality of not, which includes peer review, the journal it's published in, the research lab that wrote the paper. Reading a paper itself is a multi hour ordeal and you want to have a decent idea that it's not a crap paper before diving in. Believing what the paper says is something of a gamble because you really cannot just replicate an experiment that easily. And you need to read many many many papers before you can start doing your own research. So you want lots of assurances that a paper is good quality.

Re: Why isn't preprint review being adopted?

#92

Post by bumby »

llm_trw wrote: post #89
bumby wrote: post #85

Earlier quoted context omitted.

Necessary to share ideas? No. Necessary to share important ideas? Maybe. One of the aspects of journals that I don't see talked about much here is the curation of articles. Saying (good) journals aren't necessary is like saying journalists aren't necessary to get your news in the internet era. Not all information is necessarily good, and we often rely on systems/people to help winnow the amount we have to sift throug…

It's more like saying that maybe we don't need the NYT, the people who lied about Iraq: https://www.theguardian.com/world/2004/may/27/media.iraq , to get good quality journalism when the internet exists. Journals are in the same boat. Any good they have done is dwarfed by the few 'honest mistakes', like Alzheimer's cabal: https://www.statnews.com/2019/06/25/alzheimers-cabal-thwarte... , which have done more to stifle…

I think you're conflating the notion of perfect curation. I don't think anyone is claiming that journals or journalists are infallible. They certainly make m, although I don't think I agree that the good is "dwarfed" by them, except when used as a direct attempt to undermine credibility by adversaries.

Put differently, do you think bad ideas spread more easily with the internet? The current research seems to think so, and if you agree, I don't see how that mitigates the spread of bad information. If anything, it exacerbates it.

Re: Why isn't preprint review being adopted?

#93

Post by llm_trw »

bumby wrote: post #92
llm_trw wrote: post #89

Earlier quoted context omitted.

It's more like saying that maybe we don't need the NYT, the people who lied about Iraq: https://www.theguardian.com/world/2004/may/27/media.iraq , to get good quality journalism when the internet exists. Journals are in the same boat. Any good they have done is dwarfed by the few 'honest mistakes', like Alzheimer's cabal: https://www.statnews.com/2019/06/25/alzheimers-cabal-thwarte... , which have done more to stifle…

I think you're conflating the notion of perfect curation. I don't think anyone is claiming that journals or journalists are infallible. They certainly make m, although I don't think I agree that the good is "dwarfed" by them, except when used as a direct attempt to undermine credibility by adversaries. Put differently, do you think bad ideas spread more easily with the internet? The current research seems to think so…

I think that every bad idea spread by the internet dwarfs the harms done by ideas spread by government. Every issue with the middle east over the last 20 years is the direct result of a crusade launched on a lie enthusiastically repeated by the same people claiming the moral high ground today.

Re: Why isn't preprint review being adopted?

#94

Post by matthewdgreen »

hiddencost wrote: post #37
nitwit005 wrote: post #22

Earlier quoted context omitted.

Unfortunately, what you'll actually incentivize is spending as little effort as possible to get the money paid out.

I hear this sentiment a lot. There was a time when academia was intensely driven by culture. People did stuff because they cared about the community surviving. It is, in fact, possible to choose the "cooperate" triangle of the prisoner's dilemma, but it takes a lot of work, and is vulnerable to being wrecked by people who don't care / believe in the goals.

If you keep increasing publication requirements in a Red Queen’s race, you both increase the amount of peer-review required and also reduce the time that qualified scientists have to do it. No matter how much people want to build a functioning system, they only have so many hours in the day.

Re: Why isn't preprint review being adopted?

#95

Post by sseagull »

michaelt wrote: post #33
andyferris wrote: post #29

Earlier quoted context omitted.

It might cost $100k - $1m (or more) to repeat the work and run replications. The $5k - $10k mentioned earlier would be enough to allow time reading and thinking and checking some stuff on pen-and-paper.

> The $5k - $10k mentioned earlier would be enough to allow time reading and thinking and checking some stuff on pen-and-paper. The average postdoc in the US earns $32.81/hour, according to the results of some googling. Even taking overheads into account, $5k should cover more than a week's full time work.

In what area of science would it take only a week or two to replicate?

It might take several days to a week of literature review just to fully understand the problem. Then you might need equipment, chemicals, cultures, etc. Then depending on the area of science, doing the actual experiment could take several weeks (waiting for reactions, computer simulations, etc). Then possibly tricky analysis and statistics on top of that.

Nowadays, science is deep

Re: Why isn't preprint review being adopted?

#96

Post by paulmd »

ska wrote: post #67
geysersam wrote: post #40

Earlier quoted context omitted.

Or, research just gets published online free of charge for everyone to access, and important work will prove itself over time by being discovered, discussed and by becoming influential. If anyone wants to change something about an article (the writing, the structure of the paper, or anything else a reviewer might want to edit) they can just do it and publish a new version. If people like the new version better, good,…

> Just make it accessible. We already have that system, it's called the internet. Nothing stops you or I from putting our ideas online for all to read, comment on, update, etc. The role of the publishers, flawed as it is, has little to do with the physical cost of producing or providing an article, and is filling (one can argue badly) a role in curation and archival that is clearly needed. Any proposal to change the…

> We already have that system, it's called the internet. Nothing stops you or I from putting our ideas online for all to read, comment on, update, etc.

Are you asking how arxiv is different from blogspot?

Re: Why isn't preprint review being adopted?

#97

Post by ska »

paulmd wrote: post #96
ska wrote: post #67

Earlier quoted context omitted.

> Just make it accessible. We already have that system, it's called the internet. Nothing stops you or I from putting our ideas online for all to read, comment on, update, etc. The role of the publishers, flawed as it is, has little to do with the physical cost of producing or providing an article, and is filling (one can argue badly) a role in curation and archival that is clearly needed. Any proposal to change the…

> We already have that system, it's called the internet. Nothing stops you or I from putting our ideas online for all to read, comment on, update, etc. Are you asking how arxiv is different from blogspot?

No, although the mechanism for hosting the content aren’t that important.

Preprint servers are very useful but haven’t replaced journals for good reasons.

Re: Why isn't preprint review being adopted?

#98

Post by rhelz »

heikkilevanto wrote: post #62
bumby wrote: post #14

Earlier quoted context omitted.

I don't disagree with the proposed idea of paying for review, but I would prefer also to have guardrails to ensure a good review. I would be willing to pay for a good review because it makes the paper/investigation better. But let's face it: under the current paradigm, there are also a lot of really bad reviews. It's one thing when it's apparent that a reviewer doesn't understand something because of a lack of clarit…

So, we just need a meta-review to review the reviews. At a cost, of course. And in order to keep that honest, we need a meta-meta-review...

chuckle recall, a paper is given to 3 or 4 reviewers. No need for hierarchies of reviewers; it’s more like a jury; if all the reviewers more or less come to the same conclusion we can have a high confidence that the decision is the correct one.

Under the proposed plan, if one of the reviewers gave a review which was radically different, or otherwise obviously slap-dash job, payment could be withheld and another reviewer commissioned.

Re: Why isn't preprint review being adopted?

#99

Post by rhelz »

bumby wrote: post #84
rhelz wrote: post #63

Earlier quoted context omitted.

Whether or not your papers pass peer review—-and which journals it is published in—are important criteria for hiring, tenure, whether your grants are funded, etc. If you get rid of peer review, it’s not science. It’s just a vanity press.

> If you get rid of peer review, it’s not science. It’s just a vanity press. If you believe replicability is central to science, the current paradigm doesn't necessarily converge on science either. And when people are graded on how many publications they garner, it borders on turning publication into a symbol of status rather than one of science.

> If you believe replicability is central to science.

I do believe that, but it doesn’t matter what anybody believes, replicatable experiments and results, which peers can review and agree on, are the soul of science.

Without that it’s not science, it’s just creative writing.

Re: Why isn't preprint review being adopted?

#100

Post by timr »

pacbard wrote: post #9
timr wrote: post #2

Even if academics could review all the papers on a preprint server (which the article argues -- rightly -- that they can't), it wouldn't solve the perceived problems (or the actual problems) with scientific peer review. The vast majority of irreproducible papers aren't detectible as irreproducible at time of publication. They look fine, and many actually are fine . They just don't reproduce. That's an expected outcom…

To your second point, I alway go back to this quote: "You can't fix by analysis what you bungled by design" (Light, Singer and Willett, 1990). If a paper is broken by design, there isn't much to do after the fact. It's just broken. The problem is that doing a good RCT takes both time and effort, with the huge risk of having null results, which usually results in a desk rejection from most top journals. So, you either…

> Releasing the data won't really help much if the data generating process is flawed. Sure, other people will be able to run different kind of analyses (e.g., jackknife your standard errors instead of just using a robust correction), but I'm not sure how helpful that will be.

It allows motivated people to catch more subtle forms of nonsense. Data colada, for example, has caught outright fraud, but only through herculean efforts. Imagine what groups like this might do if they had the raw data.

Post reply on HN