Let’s make peer review scientific
nature.com
Let’s make peer review scientific
1–10 of 146 posts
Re: Let’s make peer review scientific
#2This issue of scientists being OK with totally non-scientific processes is all too common. When I was an academic my Department used to make an enormous fuss about the year to year jitter in student teaching evaluations. My colleagues (all scientists) would sit around discussing why they were a heroes because their evaluation went up 10% from last year or what they had to change because it went down 10%. I used to just sit there thinking if this sort of analysis was in a paper they were reviewing they would have ripped the authors to shreds.
Peer review fails on all levels. It does not functions as a source of quality control (everything gets published eventually) and even worse it rarely improves the quality of the paper being reviewed. I have published dozens of papers over the years and on only one occasion has the review process improved the paper - in most cases the reviewers demands made the papers worse by forcing me to remove important information or include irrelevant details (citing the reviewers publications mostly).
Re: Let’s make peer review scientific
#3Perhaps it's a question of incentives. What exactly do you get out of reading a quite complex piece of work and saying your opinion?
The closest I can come, as a non-academic, is perhaps reading other people's code.
It's bloody hard. You need to concentrate, and it's not like reading a newspaper article at all. Even small errors are not easy to spot, and it's even hard to know whether the code is structured in the way the comments say.
It somewhat makes sense to do the exercise if I have to be using the codebase. If I'm just commenting and giving a thumbs up/down, it's quite easy to reduce effort, come up with some generic comments, and see how the other reviewers do. Which is a recipe for letting errors through.
Re: Let’s make peer review scientific
#4This issue of scientists being OK with totally non-scientific processes is all too common. When I was an academic my Department used to make an enormous fuss about the year to year jitter in student teaching evaluations. My colleagues (all scientists) would sit around discussing why they were a heroes because their evaluation went up 10% from last year or what they had to change because it went down 10%. I used to ju…
I think that example is a bit unfair, teaching really isn't scientific when you get to the core of it. You cannot expect professors to be able to conduct double blind studies with student grades and so a more overarching non-scientific discussion is probably more amenable to actual improvement in performance.
Re: Let’s make peer review scientific
#5Hypothesis: peer-reviewed articles aren't rigorous
Prediction: a flawed article will pass peer-review
Testing: https://svpow.com/2013/10/03/john-bohannons-peer-review-stin...
Analysis: Peer review is not rigorous
Re: Let’s make peer review scientific
#6This issue of scientists being OK with totally non-scientific processes is all too common. When I was an academic my Department used to make an enormous fuss about the year to year jitter in student teaching evaluations. My colleagues (all scientists) would sit around discussing why they were a heroes because their evaluation went up 10% from last year or what they had to change because it went down 10%. I used to ju…
Peer review is broken not because the process of review is broken but because the peers are broken. Reviewers, like all academics, have screwed up incentives to get there name on as many papers as possible and those papers cited widely, regardless of the quality of the science involved. They will always find a way game the system to their benefit.
Recently heard about reviewers of a friend's paper blocking publication until the reviewer could publish his research on the same topic first. The whole system is just too screwed up to be fixed.
Re: Let’s make peer review scientific
#7This issue of scientists being OK with totally non-scientific processes is all too common. When I was an academic my Department used to make an enormous fuss about the year to year jitter in student teaching evaluations. My colleagues (all scientists) would sit around discussing why they were a heroes because their evaluation went up 10% from last year or what they had to change because it went down 10%. I used to ju…
I think that example is a bit unfair, teaching really isn't scientific when you get to the core of it. You cannot expect professors to be able to conduct double blind studies with student grades and so a more overarching non-scientific discussion is probably more amenable to actual improvement in performance.
You can actually conduct scientific studies on teaching and grading, but that is not the point I was making. What concerned me is how a department full of scientists wasted their time discussing the importance of results which were statistically meaningless. The scatter was just noise yet we wasted hours (days) of our time discussing the supposed importance of this noise.
Re: Let’s make peer review scientific
#8I'm enclined to think that an open review process by using a web system with a space for open discussion/commenting could be a good system. This wasn't possible before the web.
Actual system is publish or not, but it is possible now to publish everything and use a rating system. This removes the risk of plagiarism and anteriority disputes.
Re: Let’s make peer review scientific
#9This issue of scientists being OK with totally non-scientific processes is all too common. When I was an academic my Department used to make an enormous fuss about the year to year jitter in student teaching evaluations. My colleagues (all scientists) would sit around discussing why they were a heroes because their evaluation went up 10% from last year or what they had to change because it went down 10%. I used to ju…
Peer review is broken not because the process of review is broken but because the peers are broken. Reviewers, like all academics, have screwed up incentives to get there name on as many papers as possible and those papers cited widely, regardless of the quality of the science involved. They will always find a way game the system to their benefit. Recently heard about reviewers of a friend's paper blocking publicatio…
Yes this sort of bad thing happens, but more in the real high stakes games of grant reviews than publications. You can easily kill the grant of one of your competitors by just giving it a mediocre review (without any basis of course). I would see this happen all the time to my colleagues where they would get a spiteful review on a grant that would kill it and there was nothing they could do.
Re: Let’s make peer review scientific
#10We could allow everything to be published and then use a continual monitoring approach. After all the limiting resource these days is not paper but reviewers time, so lets distribute that. What is wrong with model like HN uses for open review and comment on work? Even a voting system to help identify the most 'important' works.
Edit: Haha great minds chmike