Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let’s make peer review scientific

nature.com

Post reply on HN
Print view View original

111–120 of 146 posts

Re: Let’s make peer review scientific

#111

Post by apathy »

splatcollision wrote: post #17
chmike wrote: post #8

I'm enclined to think that an open review process by using a web system with a space for open discussion/commenting could be a good system. This wasn't possible before the web. Actual system is publish or not, but it is possible now to publish everything and use a rating system. This removes the risk of plagiarism and anteriority disputes.

Had the same kind of idea - apparently "Facebook for Scientists" is apparently a thing. http://www.sciencemag.org/careers/2014/08/researchgate-faceb... https://www.researchgate.net/

researchgate is a fucking cancer. Dark patterns rule there

Re: Let’s make peer review scientific

#112

Post by apathy »

Fomite wrote: post #85
yayday wrote: post #68

Earlier quoted context omitted.

You're already moving your goalposts. Your first post said replicates the finding. If I simply post my results and someone else looks at the data and agrees with me then are they really replicating my finding or did they, oh what's the word… peer review my work and agreed?

I really dislike the idea that "Your code runs on your data" is a form of replication.

It's a minimum standard and one that many published works cannot meet. Universal enforcement of this minimum would itself be a big step forward.

Re: Let’s make peer review scientific

#113

Post by danieltillett »

jsprogrammer wrote: post #72
danieltillett wrote: post #30

Earlier quoted context omitted.

I am in Australia where the pool of potential grant reviewers is pretty small for most areas. When you have a grant success rate of 15% it is all too easy to nobble your competitors via the review process.

Why is science viewed as a competition there?

It is a competitive everywhere, just in Australia you are playing in a much smaller pool. There just is not enough money to fund all good grant applications.

Re: Let’s make peer review scientific

#114

Post by apathy »

Declanomous wrote: post #62
mariusz79 wrote: post #21

Earlier quoted context omitted.

it will, if you require that the data is shared and can be analyzed by another group. In some cases it will slow down the progress initially, but I think in the long term we would be getting much better results.

You can fabricate data nearly as easily as you can fabricate results. It is easier to detect fabricated results from raw data than it is from the final results, but fraud adapts to the methods used to detect it. If you create a new way to detect fraud, fraud will change to avoid that method. Detecting fraud is an arms race.

At some point it becomes easier to just run the damned experiment. Similar to buying a safe based on making its contents not worth the trouble to crack it.

This would be a good first step. "Photoshop for biologists" (as a workshop) is very much a thing.

Re: Let’s make peer review scientific

#115

Post by apathy »

djokkataja wrote: post #71
Jabbles wrote: post #19

Earlier quoted context omitted.

That's not going to work for Astronomy, or results of multi-year studies.

Long-term studies or not, it would be useful to have an up-to-date system that clearly indicated whether or not the results of some particular study had been replicated, and how many times they had been replicated if at all, and if there were any conflicting results between studies.

That is what Cochrane does in the UK. And indeed it is great.

Re: Let’s make peer review scientific

#116

Post by Fomite »

apathy wrote: post #112
Fomite wrote: post #85

Earlier quoted context omitted.

I really dislike the idea that "Your code runs on your data" is a form of replication.

It's a minimum standard and one that many published works cannot meet. Universal enforcement of this minimum would itself be a big step forward.

It is a minimum standard, and a useful one. It mostly annoys me when certain groups of advocates seem to treat it as an end in and of itself, and evidence that something is "repeatable" and will end the problems in obtaining scientific evidence.

I come from a field where something isn't a reproduced until it's also found in an entirely different study on a different population.

Re: Let’s make peer review scientific

#117

Post by danieltillett »

xioxox wrote: post #48
danieltillett wrote: post #29

Earlier quoted context omitted.

I am curious as to what improvements the reviewers made to your papers? What areas did they improve? In the case of the one paper of mine that was improved one of the reviewers noticed a connection between our data and a paper published 10 years before in a different field that made our work much more interesting. Of course after incorporating this the editor rejected the paper despite all three reviewers saying it s…

The suggestions have included alternative explanations for the results, useful checks on the data analysis procedures, sensible ways to restructure a paper, useful plots which could be included, places where the text was unclear, indicating where our claims were too strong, how to strengthen the paper by comparisons with theory. One referee's questions were useful to reveal that an undergraduate's paper (of which I w…

I wish that was the way the biological sciences worked.

My personal favourite peer review process story was from one of my first papers where I wrote up the whole thing and my supervisor added his name and sent it off. This was in the pre-electronic days where you physically posted off the manuscript. Serveral months when by and we heard nothing. After around six months of silence my supervisor decides to ring the editor and find out what is happening (proactive he was not). The editor tells him "oh didn't you know that paper was published a couple of months ago" - not a single change made or reviewers comment to deal with. I wish all my subsequent papers had gone like that :)

Re: Let’s make peer review scientific

#118

Post by apathy »

beevai142 wrote: post #102
apathy wrote: post #94

Earlier quoted context omitted.

I take it you're not familiar with "crowdfunding" sources like the AACR, LLS, ASCO, or other professional societies? As someone who is funded by several of the above, and who noted that their review processes were substantially less bullshit-intensive yet no less rigorous than NIH review (which has many benefits, efficiency not among them), I'm going to go out on a limb and suggest that it's possible. As far as the N…

The difference between professional societies and crowdfunding is that professionals, not the crowd who donate directly, decides which projects to fund. In this sense, I do not see a great qualitative different to government funding agencies --- if you do, please elaborate. EDIT: And to clarify, in the societies I know, general members do not directly take part in grant decision processes. Rather, the decisions are m…

It's different from crowdsourcing, but the source and sink for the funds also tend to be more closely related. Ultimately I don't really believe that major initiatives (eg P01-level grants) can be adequately reviewed by anything other than genuine peers.

But by the same token, an exploratory study requesting $30k for field work or sample processing could very well be evaluated by less skilled peers. Actually, I think I'm going to try and shop this to a friend at NIH. I'll fail, most likely, but at least I won't just be whining.

For example, pharma and big donors use the LLS review system as a "study section lite" to hand out grants larger than a typical R01. The paperwork and BS isn't really necessary at that level and just gets in the way. If something like this existed for "lark" projects, inside or outside of NIH/NSF, perhaps more diverse and potentially diversifying proposals would be worth submitting.

Re: Let’s make peer review scientific

#119

Post by apathy »

beevai142 wrote: post #102
apathy wrote: post #94

Earlier quoted context omitted.

I take it you're not familiar with "crowdfunding" sources like the AACR, LLS, ASCO, or other professional societies? As someone who is funded by several of the above, and who noted that their review processes were substantially less bullshit-intensive yet no less rigorous than NIH review (which has many benefits, efficiency not among them), I'm going to go out on a limb and suggest that it's possible. As far as the N…

The difference between professional societies and crowdfunding is that professionals, not the crowd who donate directly, decides which projects to fund. In this sense, I do not see a great qualitative different to government funding agencies --- if you do, please elaborate. EDIT: And to clarify, in the societies I know, general members do not directly take part in grant decision processes. Rather, the decisions are m…

The paperwork burden for an NIH grant (relative to a society grant) is often a large scalar multiple. The accountability is often on a par with, or less than, the typical society grant. It mystifies me why this should be so.

Re: Let’s make peer review scientific

#120

Post by apathy »

Fomite wrote: post #116
apathy wrote: post #112

Earlier quoted context omitted.

It's a minimum standard and one that many published works cannot meet. Universal enforcement of this minimum would itself be a big step forward.

It is a minimum standard, and a useful one. It mostly annoys me when certain groups of advocates seem to treat it as an end in and of itself, and evidence that something is "repeatable" and will end the problems in obtaining scientific evidence. I come from a field where something isn't a reproduced until it's also found in an entirely different study on a different population.

Oh, same here. Two of my current manuscripts have now been replicated in 3-5 independent clinical cohorts. The thing is, if the field won't even recognize that "ability to produce the same outputs given your inputs" is critical, good fucking luck with things like "actually replicates in a separate population".

Neuroscience is particularly horrendous about this.

Post reply on HN