There's a peer review scandal currently underway involving a top economics journal.
Let’s make peer review scientific
61–70 of 146 posts
Re: Let’s make peer review scientific
#62Earlier quoted context omitted.
That's not going to work for Astronomy, or results of multi-year studies.
it will, if you require that the data is shared and can be analyzed by another group. In some cases it will slow down the progress initially, but I think in the long term we would be getting much better results.
You can fabricate data nearly as easily as you can fabricate results. It is easier to detect fabricated results from raw data than it is from the final results, but fraud adapts to the methods used to detect it. If you create a new way to detect fraud, fraud will change to avoid that method. Detecting fraud is an arms race.
Re: Let’s make peer review scientific
#63Shameless plug: I'm working at the MIT Media Lab on the PubPub project ( http://www.pubpub.org ), a free platform for totally open publishing designed to solve a lot of these problems: One is peer review, which, as some have already mentioned, needs to be done in an open, ongoing, and interactive forum. Making peer review transparent to both parties (and the public) makes everyone more honest. Another is the incentiv…
> [W]e need modern, digitally-native documents that are as rich as the information they contain.
It's best to separate interactive parts from the non interactive. When I read papers I always print them because I can take notes, underline, draw shapes, comment etc. all in-line with a pen that I'll buy from a closest little shop if I don't have one with me. It is more convenient that any software tool out there, because there is basically no limitation to how I'll use the pen on the (actual) paper.
But how will I be able to print a video, or a graphic that has some controls to be fiddled with?
But they can be provided separately, and if I have the mediums, I'll use also them.
Re: Let’s make peer review scientific
#64Shameless plug: I'm working at the MIT Media Lab on the PubPub project ( http://www.pubpub.org ), a free platform for totally open publishing designed to solve a lot of these problems: One is peer review, which, as some have already mentioned, needs to be done in an open, ongoing, and interactive forum. Making peer review transparent to both parties (and the public) makes everyone more honest. Another is the incentiv…
What's wrong with PDFs? PDFs are great because it's a downloadable format and when I create one I know that what I see is what other people will see. The same can not be said for a website format where people use different browsers, different devices, and different versions of different software.
I just don't see what's wrong with using PDFs. If I want to read one of the papers on your website on my phone when I don't have Internet, how do I do it? If I just keep the page open on my phone's browser it's just going to go away at some point as my phone's OS clears the browser's cache to make room for other apps that I'm actively using. With PDFs I can just open the pdf I've downloaded.
Re: Let’s make peer review scientific
#65Shameless plug: I'm working at the MIT Media Lab on the PubPub project ( http://www.pubpub.org ), a free platform for totally open publishing designed to solve a lot of these problems: One is peer review, which, as some have already mentioned, needs to be done in an open, ongoing, and interactive forum. Making peer review transparent to both parties (and the public) makes everyone more honest. Another is the incentiv…
How are you planning on addressing the fact that none of this matters in the slightest if academic career growth/incentives/reputation specifically revolves around publishing boring pdfs in established journals? What do I gain by sending my work to your platform where the first question anyone in my "target audience" will ask is "why didn't this get published in a real journal, whats wrong with it?", and I will get e…
IMO, the change has to come from the grant financiers, and a case can be made that they get a greater more bang for their buck by adding requirements such as pre-registration, open data publishing (so long as they don't want to keep the data proprietary, of course), and publishing guaranteed regardless of outcome. The fruits of what they purchased can be of more use to the field in this manner, and even they themselves may find they spend less money if the 'failures' are documented in such a way that they don't inadvertently repeatedly fund them due to the fact that the initial results were hidden away in a filing cabinet.
I think the same case could be made for placing more emphasis on funding replication -- instead of thinking 'we learned it, why revisit it?', you're checking if it's actually valuable information worth spending more on, or just a fluke.
Re: Let’s make peer review scientific
#66Shameless plug: I'm working at the MIT Media Lab on the PubPub project ( http://www.pubpub.org ), a free platform for totally open publishing designed to solve a lot of these problems: One is peer review, which, as some have already mentioned, needs to be done in an open, ongoing, and interactive forum. Making peer review transparent to both parties (and the public) makes everyone more honest. Another is the incentiv…
I agree with most of your points, but disagree strenuously with this one:
It's 2016, and these dead, flat, static PDFs are functionally identical to the paper they replaced!
There's a lot of value in static PDFs and how we format documents for publication. $CURRENT_YEAR doesn't invalidate good ideas from ($CURRENT_YEAR-n).
Just in terms of technology, static PDFs are:
- Easily distributed. - Easily archived. - Readable forever, even as platforms change incompatibly. - Readable on any device.
20 years from now, do you really want to have to run an old browser, or debug shoddy JS/HTML/whatever, just to read a paper from $CURRENT_YEAR-20?
Re: Let’s make peer review scientific
#67Earlier quoted context omitted.
I disagree. Something similar exists on the wikipedia and it's still open to biases and agendas, it's just a matter of who is more persistent and which side the admins take.
But you don't need the 'gatekeeper'. Imagine if you had a 'trustability' factor that people could cryptographically sign as themselves. You could build a 'trust web', where papers link to other papers by an author. That author's trust ratings could go up based on the citer's evidence. There's clearly tech waiting to be built here, which might or might not be a startup.
There's already a "trustability" factor: it's called a name. When I search for a new paper and find something interesting, I look at the first author, the name of the PI, and the name of the university or laboratory associated with this paper.
If I'm familiar with the work of the first author then I already have an idea of how much I can trust his/her work. If I don't know this person then I look at the coauthors and especially the PI. If the PI is a heavy hitter in my field then there's my trustability factor right there. If I don't know the PI but I know the laboratory or university it's affiliated with then there's my third line of a trustability metric.
Re: Let’s make peer review scientific
#68Earlier quoted context omitted.
That's not going to work for Astronomy, or results of multi-year studies.
it will, if you require that the data is shared and can be analyzed by another group. In some cases it will slow down the progress initially, but I think in the long term we would be getting much better results.
You're already moving your goalposts. Your first post said replicates the finding. If I simply post my results and someone else looks at the data and agrees with me then are they really replicating my finding or did they, oh what's the word… peer review my work and agreed?
Re: Let’s make peer review scientific
#69Earlier quoted context omitted.
That's not going to work for Astronomy, or results of multi-year studies.
it will, if you require that the data is shared and can be analyzed by another group. In some cases it will slow down the progress initially, but I think in the long term we would be getting much better results.
[deleted]
Re: Let’s make peer review scientific
#70Earlier quoted context omitted.
How are you planning on addressing the fact that none of this matters in the slightest if academic career growth/incentives/reputation specifically revolves around publishing boring pdfs in established journals? What do I gain by sending my work to your platform where the first question anyone in my "target audience" will ask is "why didn't this get published in a real journal, whats wrong with it?", and I will get e…
Great question, and one that certainly doesn't have a straightforward or trivial answer. It's definitely more of a social challenge than a technical one - making publishing free/open won't do anything to fix incentives on its own. My hunch is that change to this system will come from the outside. It's too risky of a career decision for a tenure-track professor to start publishing on PubPub (or any open/new system). B…
Perhaps offering some sort of crowdsourced funding mechanism and a reputation system would go a long way toward correcting some of these incentives?
For example, giving authors / organizations a Bitcoin address where they can receive funds from individuals / organizations who want to support their research.
Also, awarding reputation to authors based on the level of peer review their research has successfully undergone (number of peers, level of rigor, etc.), and conversely awarding reputation and funding to those who perform peer reviews. Allowing users to contribute to a peer review fund for individual articles or in general.
All that to say this is very exciting and opens up a lot of new possibilities.