Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let’s make peer review scientific

nature.com

Post reply on HN
Print view View original

41–50 of 146 posts

Re: Let’s make peer review scientific

#41

Post by suchsciencewow »

joelg wrote: post #20

Shameless plug: I'm working at the MIT Media Lab on the PubPub project ( http://www.pubpub.org ), a free platform for totally open publishing designed to solve a lot of these problems: One is peer review, which, as some have already mentioned, needs to be done in an open, ongoing, and interactive forum. Making peer review transparent to both parties (and the public) makes everyone more honest. Another is the incentiv…

How are you planning on addressing the fact that none of this matters in the slightest if academic career growth/incentives/reputation specifically revolves around publishing boring pdfs in established journals? What do I gain by sending my work to your platform where the first question anyone in my "target audience" will ask is "why didn't this get published in a real journal, whats wrong with it?", and I will get essentially zero resume kudos points for it? Keeping in mind that it took me at least many months to do the work, I don't exactly have an abundance of papers to throw around, and my officemate almost certainly will submit work of similar quality to a traditional journal and reap the benefits of that.

Just saying, my decision to publish in traditional journals isn't as much as a decision as it is a requirement of the very career I am attempting to pursue.

Re: Let’s make peer review scientific

#42

Post by joelg »

bixiou wrote: post #37
joelg wrote: post #20

Shameless plug: I'm working at the MIT Media Lab on the PubPub project ( http://www.pubpub.org ), a free platform for totally open publishing designed to solve a lot of these problems: One is peer review, which, as some have already mentioned, needs to be done in an open, ongoing, and interactive forum. Making peer review transparent to both parties (and the public) makes everyone more honest. Another is the incentiv…

If I might give my opinion on which features PubPub should implement next: - adding "Math in-line $$" in Formatting menu - enabling upload of LaTeX files - enabling upload of PDFs, to store on PubPub existing research (combined by via.hypothes.is, one can annotate PDFs, which is in the spirit of PubPub) - include some interesting features of SJS like grading, download as PDF, add to personal library... I am impressed…

Haha - not my project, I just work on it for now. But thanks for the great ideas!

Re: Let’s make peer review scientific

#44

Post by isTravis »

suchsciencewow wrote: post #41
joelg wrote: post #20

Shameless plug: I'm working at the MIT Media Lab on the PubPub project ( http://www.pubpub.org ), a free platform for totally open publishing designed to solve a lot of these problems: One is peer review, which, as some have already mentioned, needs to be done in an open, ongoing, and interactive forum. Making peer review transparent to both parties (and the public) makes everyone more honest. Another is the incentiv…

How are you planning on addressing the fact that none of this matters in the slightest if academic career growth/incentives/reputation specifically revolves around publishing boring pdfs in established journals? What do I gain by sending my work to your platform where the first question anyone in my "target audience" will ask is "why didn't this get published in a real journal, whats wrong with it?", and I will get e…

Great question, and one that certainly doesn't have a straightforward or trivial answer. It's definitely more of a social challenge than a technical one - making publishing free/open won't do anything to fix incentives on its own.

My hunch is that change to this system will come from the outside. It's too risky of a career decision for a tenure-track professor to start publishing on PubPub (or any open/new system). But, there are lots of people who aren't playing that game. Lots of people who are doing science outside of academia, at a corporate R&D position, or for the sake of education, etc.

The most important step is to show that open publishing works. If we can work with these early adopters and show that conversations are more rich, or results more reproducible, we can start to go to universities and grant agencies and advocate for them to require open publishing. The first day that a university hires a professor or an agency rewards a grant based on the history of openly published work, will be a turning point. I hope it will be similar to the first time a software dev was hired for their Github profile, rather than their CS degree.

Today, software companies hire on experience. A university degree can show that, but so can major contributions to an open-source project. I hope science can become the same. Whether you're a PhD out of a great program, or an high-school drop out that has committed her life to rigorous experimentation, your demonstrated experience should be what you're hired on, not the list of journals that have found it in their interest (many of them are for-profit) to include your work.

Re: Let’s make peer review scientific

#45

Post by zyxzevn »

The peer-review is mixed with a reductionistic structure. Each scientific area is specialized. The articles are reviewed by people in the specialized fields. Often they find something that is not entirely in their field of knowledge. That is because they are trying to "science" something new.

So they can produce theories that seem a solution from their field of knowledge, but are not valid in another field. The peers will not see this problem. Another problem is that engineers that try to apply this science find practical problems. But because they are seen as "of lesser knowledge" their practical criticism is often rejected.

The best example I know is the "theory" that magnetic fields can bump into each other, producing energy. In this model the fields are made of flux lines, which can bumb into other flux lines. The flux lines will then break and reorganize, producing energy.

Yet, as I write this, you may already think that this is bullshit, because flux-lines are imaginary lines to describe a magnetic field.

Now, with this in mind, look at magnetic reconnection. This is a theory made scientists specialized in astronomy, and not in the area of electromagnetism.

I believe that this problem is in every (specialized) area of science.

Re: Let’s make peer review scientific

#46

Post by kriro »

suchsciencewow wrote: post #41
joelg wrote: post #20

Shameless plug: I'm working at the MIT Media Lab on the PubPub project ( http://www.pubpub.org ), a free platform for totally open publishing designed to solve a lot of these problems: One is peer review, which, as some have already mentioned, needs to be done in an open, ongoing, and interactive forum. Making peer review transparent to both parties (and the public) makes everyone more honest. Another is the incentiv…

How are you planning on addressing the fact that none of this matters in the slightest if academic career growth/incentives/reputation specifically revolves around publishing boring pdfs in established journals? What do I gain by sending my work to your platform where the first question anyone in my "target audience" will ask is "why didn't this get published in a real journal, whats wrong with it?", and I will get e…

I think it's a political problem. Most researchers and professors tend to be employed by the state. Lobby the state to give very high preference to open access publication. In an ideal world I'd read a sentence like this in a hiring guideline for state universities: "only research that is available to all citizens of this state shall be considered as a contribution by the candidate"

Additionally for research project funding I want to read a similar sentence in the proposed outline that you can typically download for the grant proposal. "Only list your research projects and open access publications in the Previous Related Work section" (or high preference will be given to ...)

Basically state funded research that is not available to citizens should be heavily penalized. I think that is logically very consistent. In fact I'd love to GPL the process and make it a requirement to OA everything as soon as a single cent of money goes towards the research.

Re: Let’s make peer review scientific

#47

Post by freyfogle »

bixiou wrote: post #37
joelg wrote: post #20

Shameless plug: I'm working at the MIT Media Lab on the PubPub project ( http://www.pubpub.org ), a free platform for totally open publishing designed to solve a lot of these problems: One is peer review, which, as some have already mentioned, needs to be done in an open, ongoing, and interactive forum. Making peer review transparent to both parties (and the public) makes everyone more honest. Another is the incentiv…

If I might give my opinion on which features PubPub should implement next: - adding "Math in-line $$" in Formatting menu - enabling upload of LaTeX files - enabling upload of PDFs, to store on PubPub existing research (combined by via.hypothes.is, one can annotate PDFs, which is in the spirit of PubPub) - include some interesting features of SJS like grading, download as PDF, add to personal library... I am impressed…

Why reinvent this wheel? Why not instead (also) partner with people like Overleaf who already have many of these features?

https://www.overleaf.com

Re: Let’s make peer review scientific

#48

Post by xioxox »

danieltillett wrote: post #29
xioxox wrote: post #18

Earlier quoted context omitted.

> and even worse it rarely improves the quality of the paper being reviewed. That might be your experience, but I think it has significantly improved the quality of a large fraction of the papers I have been an author or co-author on. I have encountered one bad reviewer in the past, and in retrospect perhaps I should have taken more notice of his report. As a reviewer, I've only had one negative incident, which was w…

I am curious as to what improvements the reviewers made to your papers? What areas did they improve? In the case of the one paper of mine that was improved one of the reviewers noticed a connection between our data and a paper published 10 years before in a different field that made our work much more interesting. Of course after incorporating this the editor rejected the paper despite all three reviewers saying it s…

The suggestions have included alternative explanations for the results, useful checks on the data analysis procedures, sensible ways to restructure a paper, useful plots which could be included, places where the text was unclear, indicating where our claims were too strong, how to strengthen the paper by comparisons with theory.

One referee's questions were useful to reveal that an undergraduate's paper (of which I was a co-author) had some fairly serious flaws which required it was withdrawn. Fortunately this has been the only "rejection" I've suffered so far, other than papers speculatively submitted to Nature or Science which were later published in mainstream journals. In astronomy and astrophysics we are fortunate to have only 3-4 major journals with similar impact factors.

Re: Let’s make peer review scientific

#49

Post by dluan »

suchsciencewow wrote: post #41
joelg wrote: post #20

Shameless plug: I'm working at the MIT Media Lab on the PubPub project ( http://www.pubpub.org ), a free platform for totally open publishing designed to solve a lot of these problems: One is peer review, which, as some have already mentioned, needs to be done in an open, ongoing, and interactive forum. Making peer review transparent to both parties (and the public) makes everyone more honest. Another is the incentiv…

How are you planning on addressing the fact that none of this matters in the slightest if academic career growth/incentives/reputation specifically revolves around publishing boring pdfs in established journals? What do I gain by sending my work to your platform where the first question anyone in my "target audience" will ask is "why didn't this get published in a real journal, whats wrong with it?", and I will get e…

I think the answer is closely tied to funding. The whole prestige of impact factor, national academy membership, etc are basic heuristics that help the current funding process. Grant review committees comprised of already established scientists and administrators are more likely to fund what will show up in boring paywalled PDFs.

Given another way scientists could make a living while pursuing research, I think you'd see some would stop feeling pressured into publishing in the traditional way and experimenting with altmetrics, open access, open post-pub peer review.

Re: Let’s make peer review scientific

#50

Post by wapapaloobop »

altoz wrote: post #5

Hypothesis: peer-reviewed articles aren't rigorous Prediction: a flawed article will pass peer-review Testing: https://svpow.com/2013/10/03/john-bohannons-peer-review-stin... Analysis: Peer review is not rigorous

Isn't there are standard just below rigour, though, that is still worth checking for, something like 'no obvious errors within the current paradigm'?
Post reply on HN