Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let’s make peer review scientific

nature.com

Post reply on HN
Print view View original

21–30 of 146 posts

Re: Let’s make peer review scientific

#21

Post by mariusz79 »

Jabbles wrote: post #19
mariusz79 wrote: post #16

I've got a simpler solution - scientific paper should not be considered valid until another team replicates the findings. That would quickly get rid of all of the fake results, plus would weed out all of the research that nobody really cares about. Also require that all data is shared, and that until your results are not verified or replicated by a team from another university/country you don't get any more funds.

That's not going to work for Astronomy, or results of multi-year studies.

it will, if you require that the data is shared and can be analyzed by another group. In some cases it will slow down the progress initially, but I think in the long term we would be getting much better results.

Re: Let’s make peer review scientific

#22

Post by ap22213 »

chmike wrote: post #8

I'm enclined to think that an open review process by using a web system with a space for open discussion/commenting could be a good system. This wasn't possible before the web. Actual system is publish or not, but it is possible now to publish everything and use a rating system. This removes the risk of plagiarism and anteriority disputes.

Isn't 'the scientific paper' a kind of dated concept? I'd love to see the whole scientific process from start to finish in a 'notebook' form. Why summarize findings in a paper at the end of it all, when all steps (including conception, data gathering, successes, missteps, comments, and corrections) could all be viewed as a whole? It would seem to be more in line with the intent of science.

I'm not a scientist, obviously. And, I realize that practical science has as much to do with reputation, fame, and the climbing of academic hierarchies as it does with research. But, it would be so cool to see science more as an evolving, collaborative process.

Re: Let’s make peer review scientific

#23

Post by jessriedel »

chmike wrote: post #8

I'm enclined to think that an open review process by using a web system with a space for open discussion/commenting could be a good system. This wasn't possible before the web. Actual system is publish or not, but it is possible now to publish everything and use a rating system. This removes the risk of plagiarism and anteriority disputes.

Physicists have access to this if they combine the arXiv with Scirate.

https://scirate.com/

Very few choose to, and very little discussion actually happens. There is something of a chicken-or-egg problem, but the site is sufficiently established in one community (quantum info) that if folks wanted to discuss papers they could. So I take it as weak evidence that they just don't have the incentives to do so.

If a central discussion forum were anointed as the One True Forum, I weakly predict physicists would spend time defending themselves from "drive-by criticism" (i.e., criticism that is vague and takes little thought, but consumes large resources to defend against) but little constructive would get done.

Re: Let’s make peer review scientific

#24

Post by p4bl0 »

joelg wrote: post #20

Shameless plug: I'm working at the MIT Media Lab on the PubPub project ( http://www.pubpub.org ), a free platform for totally open publishing designed to solve a lot of these problems: One is peer review, which, as some have already mentioned, needs to be done in an open, ongoing, and interactive forum. Making peer review transparent to both parties (and the public) makes everyone more honest. Another is the incentiv…

Hi! I really think what you are doing here is great and I hope you will succeed!

Is there a way to test the platform without creating to much garbage? Like a sandbox version maybe?

Re: Let’s make peer review scientific

#25

Post by joelg »

p4bl0 wrote: post #24
joelg wrote: post #20

Shameless plug: I'm working at the MIT Media Lab on the PubPub project ( http://www.pubpub.org ), a free platform for totally open publishing designed to solve a lot of these problems: One is peer review, which, as some have already mentioned, needs to be done in an open, ongoing, and interactive forum. Making peer review transparent to both parties (and the public) makes everyone more honest. Another is the incentiv…

Hi! I really think what you are doing here is great and I hope you will succeed! Is there a way to test the platform without creating to much garbage? Like a sandbox version maybe?

Everything is open-sourced, and we've tried to document as much as possible to make it easy for people to host their own instances! Alternatively, you can just create arbitrarily many drafts without publishing any of them.

https://github.com/pubpub/pubpub

Re: Let’s make peer review scientific

#26

Post by jessriedel »

joelg wrote: post #20

Shameless plug: I'm working at the MIT Media Lab on the PubPub project ( http://www.pubpub.org ), a free platform for totally open publishing designed to solve a lot of these problems: One is peer review, which, as some have already mentioned, needs to be done in an open, ongoing, and interactive forum. Making peer review transparent to both parties (and the public) makes everyone more honest. Another is the incentiv…

So I'm very interested in going beyond PDFs, but I can't for the life of me figure out what PubPub intends to do. I get that there's an online collaboration engine, as is offered by many places like Authorea, ShareLaTeX, Overleaf, and FindusWriter. What does PubPub do differently here?

Re: Let’s make peer review scientific

#27

Post by bixiou »

joelg wrote: post #20

Shameless plug: I'm working at the MIT Media Lab on the PubPub project ( http://www.pubpub.org ), a free platform for totally open publishing designed to solve a lot of these problems: One is peer review, which, as some have already mentioned, needs to be done in an open, ongoing, and interactive forum. Making peer review transparent to both parties (and the public) makes everyone more honest. Another is the incentiv…

This is great job! Congrats !!!!

Re: Let’s make peer review scientific

#28

Post by p4bl0 »

joelg wrote: post #25
p4bl0 wrote: post #24

Earlier quoted context omitted.

Hi! I really think what you are doing here is great and I hope you will succeed! Is there a way to test the platform without creating to much garbage? Like a sandbox version maybe?

Everything is open-sourced, and we've tried to document as much as possible to make it easy for people to host their own instances! Alternatively, you can just create arbitrarily many drafts without publishing any of them. https://github.com/pubpub/pubpub

Running my own instance is a bit heavy just to try the platform :). But thanks for the tips, I can create draft and delete them without trashing your instance.

The fact that it is open source is a major positive point. I suppose you have heard of SJS [1] which in a way is very close to what PubPub is, except I do not believe in SJS because of it being closed source: it is way too unsafe for a scientific community to take the risk of adopting such a media for a new journal or a conference as the platform is not open and there is a risk of Elsevier buying it to shut it down or closing it even more, for instance.

[1] http://sjscience.org/

Re: Let’s make peer review scientific

#29

Post by danieltillett »

xioxox wrote: post #18
danieltillett wrote: post #2

This issue of scientists being OK with totally non-scientific processes is all too common. When I was an academic my Department used to make an enormous fuss about the year to year jitter in student teaching evaluations. My colleagues (all scientists) would sit around discussing why they were a heroes because their evaluation went up 10% from last year or what they had to change because it went down 10%. I used to ju…

> and even worse it rarely improves the quality of the paper being reviewed. That might be your experience, but I think it has significantly improved the quality of a large fraction of the papers I have been an author or co-author on. I have encountered one bad reviewer in the past, and in retrospect perhaps I should have taken more notice of his report. As a reviewer, I've only had one negative incident, which was w…

I am curious as to what improvements the reviewers made to your papers? What areas did they improve?

In the case of the one paper of mine that was improved one of the reviewers noticed a connection between our data and a paper published 10 years before in a different field that made our work much more interesting. Of course after incorporating this the editor rejected the paper despite all three reviewers saying it should be published.

The editor of the next journal I sent the paper to was happy especially as the paper has since picked up over 1000 citations, but such is the vagaries of peer review.

Re: Let’s make peer review scientific

#30

Post by danieltillett »

dangerlibrary wrote: post #12
danieltillett wrote: post #9

Earlier quoted context omitted.

Yes this sort of bad thing happens, but more in the real high stakes games of grant reviews than publications. You can easily kill the grant of one of your competitors by just giving it a mediocre review (without any basis of course). I would see this happen all the time to my colleagues where they would get a spiteful review on a grant that would kill it and there was nothing they could do.

At the NIH, at least, I know that enormous effort is put into ensuring that there are not conflicts of interest between the reviewers and those being reviewed. I'm not particularly surprised to hear that some places do not put in that effort - it is a slow, painstaking process - but there are places where this is less of a problem.

I am in Australia where the pool of potential grant reviewers is pretty small for most areas. When you have a grant success rate of 15% it is all too easy to nobble your competitors via the review process.
Post reply on HN