I personally prefer reproducibility over peer review.
Let’s make peer review scientific
11–20 of 146 posts
Re: Let’s make peer review scientific
#12Earlier quoted context omitted.
Peer review is broken not because the process of review is broken but because the peers are broken. Reviewers, like all academics, have screwed up incentives to get there name on as many papers as possible and those papers cited widely, regardless of the quality of the science involved. They will always find a way game the system to their benefit. Recently heard about reviewers of a friend's paper blocking publicatio…
Yes this sort of bad thing happens, but more in the real high stakes games of grant reviews than publications. You can easily kill the grant of one of your competitors by just giving it a mediocre review (without any basis of course). I would see this happen all the time to my colleagues where they would get a spiteful review on a grant that would kill it and there was nothing they could do.
At the NIH, at least, I know that enormous effort is put into ensuring that there are not conflicts of interest between the reviewers and those being reviewed.
I'm not particularly surprised to hear that some places do not put in that effort - it is a slow, painstaking process - but there are places where this is less of a problem.
Re: Let’s make peer review scientific
#13I'm enclined to think that an open review process by using a web system with a space for open discussion/commenting could be a good system. This wasn't possible before the web. Actual system is publish or not, but it is possible now to publish everything and use a rating system. This removes the risk of plagiarism and anteriority disputes.
I disagree. Something similar exists on the wikipedia and it's still open to biases and agendas, it's just a matter of who is more persistent and which side the admins take.
Re: Let’s make peer review scientific
#14I'm enclined to think that an open review process by using a web system with a space for open discussion/commenting could be a good system. This wasn't possible before the web. Actual system is publish or not, but it is possible now to publish everything and use a rating system. This removes the risk of plagiarism and anteriority disputes.
I disagree. Something similar exists on the wikipedia and it's still open to biases and agendas, it's just a matter of who is more persistent and which side the admins take.
But you don't need the 'gatekeeper'. Imagine if you had a 'trustability' factor that people could cryptographically sign as themselves. You could build a 'trust web', where papers link to other papers by an author. That author's trust ratings could go up based on the citer's evidence.
There's clearly tech waiting to be built here, which might or might not be a startup.
Re: Let’s make peer review scientific
#15I'm enclined to think that an open review process by using a web system with a space for open discussion/commenting could be a good system. This wasn't possible before the web. Actual system is publish or not, but it is possible now to publish everything and use a rating system. This removes the risk of plagiarism and anteriority disputes.
I disagree. Something similar exists on the wikipedia and it's still open to biases and agendas, it's just a matter of who is more persistent and which side the admins take.
Yeah, it also means minorities can be silenced via the majority. While this is good for protecting against incorrect/poorly researched work, it can also prevent novel ideas from gaining traction. Though, this problem exists in the system we have now.
Re: Let’s make peer review scientific
#16I've got a simpler solution - scientific paper should not be considered valid until another team replicates the findings. That would quickly get rid of all of the fake results, plus would weed out all of the research that nobody really cares about. Also require that all data is shared, and that until your results are not verified or replicated by a team from another university/country you don't get any more funds.
Re: Let’s make peer review scientific
#17I'm enclined to think that an open review process by using a web system with a space for open discussion/commenting could be a good system. This wasn't possible before the web. Actual system is publish or not, but it is possible now to publish everything and use a rating system. This removes the risk of plagiarism and anteriority disputes.
Had the same kind of idea - apparently "Facebook for Scientists" is apparently a thing.
http://www.sciencemag.org/careers/2014/08/researchgate-faceb...
Re: Let’s make peer review scientific
#18This issue of scientists being OK with totally non-scientific processes is all too common. When I was an academic my Department used to make an enormous fuss about the year to year jitter in student teaching evaluations. My colleagues (all scientists) would sit around discussing why they were a heroes because their evaluation went up 10% from last year or what they had to change because it went down 10%. I used to ju…
> and even worse it rarely improves the quality of the paper being reviewed.
That might be your experience, but I think it has significantly improved the quality of a large fraction of the papers I have been an author or co-author on. I have encountered one bad reviewer in the past, and in retrospect perhaps I should have taken more notice of his report. As a reviewer, I've only had one negative incident, which was when my comments were overridden by the editor. My experience might be specific in my subject, as we only have one referee per paper.
Re: Let’s make peer review scientific
#19I've got a simpler solution - scientific paper should not be considered valid until another team replicates the findings. That would quickly get rid of all of the fake results, plus would weed out all of the research that nobody really cares about. Also require that all data is shared, and that until your results are not verified or replicated by a team from another university/country you don't get any more funds.
That's not going to work for Astronomy, or results of multi-year studies.
Re: Let’s make peer review scientific
#20Shameless plug: I'm working at the MIT Media Lab on the PubPub project (http://www.pubpub.org), a free platform for totally open publishing designed to solve a lot of these problems:
One is peer review, which, as some have already mentioned, needs to be done in an open, ongoing, and interactive forum. Making peer review transparent to both parties (and the public) makes everyone more honest.
Another is the incentive of publication itself as the ultimate goal. Instead, we need to think of documents as evolving, growing bodies of knowledge and compilations of ongoing research. Every step of the scientific process is important, yet most of it is flattened and compressed and lost, like most negative results, which are ditched in search of sexy click-bait headliner results.
Another is the role of publishers as gatekeepers and arbiters of truth. We need a medium in which anyone can curate a journal, and in which submission, review, and acceptance procedures are consistent and transparent.
Another is the nature of the medium itself. It's 2016, and these dead, flat, static PDFs are functionally identical to the paper they replaced! Insert your favorite Bret Victor/Ted Nelson rant here: we need modern, digitally-native documents that are as rich as the information they contain.
Another is reproduciblity. We should be able to see the code that transformed the raw dataset, tweak it, and publish our own fork, while automatically keeping the thread of attribution.
The list goes on and on...