Let’s make peer review scientific • Hacker News Board

Re: Let’s make peer review scientific

#101

Post by jsprogrammer » Wed, Jul 06, 2016, 9:03 PM UTC

chmike wrote: post #56

jessriedel wrote: post #23

Earlier quoted context omitted.

Physicists have access to this if they combine the arXiv with Scirate. https://scirate.com/ Very few choose to, and very little discussion actually happens. There is something of a chicken-or-egg problem, but the site is sufficiently established in one community (quantum info) that if folks wanted to discuss papers they could. So I take it as weak evidence that they just don't have the incentives to do so. If a centr…

arxiv is not open. At least not as open as it should be in my opinion. But I fully agree on the chicken-or-egg problem. The main problem is bootsrapping such web service. I don't think that "anoiting" a forum/web service would be a good strategy. We need to keep open the possibility of a new and better system to take over. My feeling is that two fundamental requirements must be satisfied. 1. Anybody should be able to…

One could start with a model similar to that in which most people using systems which this new system is meant to replace already find themselves.

You should be able to select peers (though, maybe you should be given some as well?) and "experts" to follow. You would only see their activity when you want to, but you could also participate in the larger network. As a practical matter of implementation, I think allowing groups to run segregated networks (ie. their own "node") which can later be joined to each other makes sense for getting actual adoption and for reducing development, infrastructure, and support costs.

I think there are likely many problems with such a crude model, but that is where we find ourselves it seems. Waiting to think up something better might be prudent, but that might be difficult without trying something out.

Re: Let’s make peer review scientific

#102

Post by beevai142 » Wed, Jul 06, 2016, 9:08 PM UTC

apathy wrote: post #94

kd0amg wrote: post #82

Earlier quoted context omitted.

> or NSF (whose yearly funding is dwarfed by the R&D funding at a single company) If the worst thing you can say about the NSF is that they need more money, that makes it sound like GP has come up with a nice way to allocate the available funding towards particular research projects. > It would be nice if you thought a little more carefully about what you wrote. The devil is in the details. Details like how to get "c…

I take it you're not familiar with "crowdfunding" sources like the AACR, LLS, ASCO, or other professional societies? As someone who is funded by several of the above, and who noted that their review processes were substantially less bullshit-intensive yet no less rigorous than NIH review (which has many benefits, efficiency not among them), I'm going to go out on a limb and suggest that it's possible. As far as the N…

The difference between professional societies and crowdfunding is that professionals, not the crowd who donate directly, decides which projects to fund. In this sense, I do not see a great qualitative different to government funding agencies --- if you do, please elaborate.

EDIT: And to clarify, in the societies I know, general members do not directly take part in grant decision processes. Rather, the decisions are made by a small panel, possibly together with external reviewers. This is fairly different from crowdsourcing.

Re: Let’s make peer review scientific

#103

Post by [deleted] » Wed, Jul 06, 2016, 9:41 PM UTC

beevai142 wrote: post #102

apathy wrote: post #94

Earlier quoted context omitted.

I take it you're not familiar with "crowdfunding" sources like the AACR, LLS, ASCO, or other professional societies? As someone who is funded by several of the above, and who noted that their review processes were substantially less bullshit-intensive yet no less rigorous than NIH review (which has many benefits, efficiency not among them), I'm going to go out on a limb and suggest that it's possible. As far as the N…

The difference between professional societies and crowdfunding is that professionals, not the crowd who donate directly, decides which projects to fund. In this sense, I do not see a great qualitative different to government funding agencies --- if you do, please elaborate. EDIT: And to clarify, in the societies I know, general members do not directly take part in grant decision processes. Rather, the decisions are m…

[deleted]

Re: Let’s make peer review scientific

#104

Post by apathy » Wed, Jul 06, 2016, 9:56 PM UTC

beevai142 wrote: post #102

apathy wrote: post #94

Earlier quoted context omitted.

I take it you're not familiar with "crowdfunding" sources like the AACR, LLS, ASCO, or other professional societies? As someone who is funded by several of the above, and who noted that their review processes were substantially less bullshit-intensive yet no less rigorous than NIH review (which has many benefits, efficiency not among them), I'm going to go out on a limb and suggest that it's possible. As far as the N…

The difference between professional societies and crowdfunding is that professionals, not the crowd who donate directly, decides which projects to fund. In this sense, I do not see a great qualitative different to government funding agencies --- if you do, please elaborate. EDIT: And to clarify, in the societies I know, general members do not directly take part in grant decision processes. Rather, the decisions are m…

To some (fairly large, in the case of ASCO or ASH or AACR, perhaps smaller for LLS or AHA) degree, the dues-paying professionals in these societies are the crowd. I would say they are a middle ground between something like an experiment.com or similar at one extreme, and NIH (which has inordinate purely political input -- ask your program officer!) at the other.

We shan't discuss scams like Komen here, but genuine research foundations can exist along a continuum.

Re: Let’s make peer review scientific

#105

Post by apathy » Wed, Jul 06, 2016, 9:59 PM UTC

untilHellbanned wrote: post #52

suchsciencewow wrote: post #41

Earlier quoted context omitted.

How are you planning on addressing the fact that none of this matters in the slightest if academic career growth/incentives/reputation specifically revolves around publishing boring pdfs in established journals? What do I gain by sending my work to your platform where the first question anyone in my "target audience" will ask is "why didn't this get published in a real journal, whats wrong with it?", and I will get e…

As an academic in biomedical sciences and as someone who tried to create a journal that did all the things the OP said, thank you. The old crappy system is mandatory. You can't decide to do otherwise and still have a job. The impetus for change needs to start at the funding source level, NIH. They need to make a journal that one is required to publish in order to keep your funding.

They have one (several, actually). The trouble is that they're called Cell, Nature, and Science; they're all for-profit; and they answer to no one.

The Wellcome Trust is walking the walk that NIH talked for the past decade. We will see how this goes. Cross fingers.

Re: Let’s make peer review scientific

#106

Post by semi-extrinsic » Wed, Jul 06, 2016, 10:04 PM UTC

joelg wrote: post #20

Shameless plug: I'm working at the MIT Media Lab on the PubPub project ( http://www.pubpub.org ), a free platform for totally open publishing designed to solve a lot of these problems: One is peer review, which, as some have already mentioned, needs to be done in an open, ongoing, and interactive forum. Making peer review transparent to both parties (and the public) makes everyone more honest. Another is the incentiv…

> Another is the nature of the medium itself. It's 2016, and these dead, flat, static PDFs are functionally identical to the paper they replaced!

Well, many of the existing publishers (at least ACS, Nature group, Wiley, Science group) are already pushing hard towards something "more" than PDF, mainly for DRM reasons. And scientists in general are resisting greatly, because PDF is such an established format, supported on all devices and platforms, can be created by the tools we use, and can be shared freely amongst peers. The barrier to adoption for a new format is extremely huge.

Re: Let’s make peer review scientific

#107

Post by apathy » Wed, Jul 06, 2016, 10:04 PM UTC

yayday wrote: post #64

joelg wrote: post #20

Shameless plug: I'm working at the MIT Media Lab on the PubPub project ( http://www.pubpub.org ), a free platform for totally open publishing designed to solve a lot of these problems: One is peer review, which, as some have already mentioned, needs to be done in an open, ongoing, and interactive forum. Making peer review transparent to both parties (and the public) makes everyone more honest. Another is the incentiv…

What's wrong with PDFs? PDFs are great because it's a downloadable format and when I create one I know that what I see is what other people will see. The same can not be said for a website format where people use different browsers, different devices, and different versions of different software. I just don't see what's wrong with using PDFs. If I want to read one of the papers on your website on my phone when I don'…

any standardized downloadable format (to include VMs or AMIs) also fits these criteria. PDFs aren't inherently bad, but on occasion they are not the best tool to communicate results.

For example, Sean Morrison's recent paper on live-imaging stem cells in vivo benefited greatly from inclusion of movies showing the live-processed data. Their point was that live adult stem cells can be imaged, and this revealed niche biology. The only way to support this was to provide the proof, and MPEG did this nicely.

Agreed regarding permanent vs ephemeral, though.

Re: Let’s make peer review scientific

#108

Post by apathy » Wed, Jul 06, 2016, 10:09 PM UTC

quantumhobbit wrote: post #6

danieltillett wrote: post #2

This issue of scientists being OK with totally non-scientific processes is all too common. When I was an academic my Department used to make an enormous fuss about the year to year jitter in student teaching evaluations. My colleagues (all scientists) would sit around discussing why they were a heroes because their evaluation went up 10% from last year or what they had to change because it went down 10%. I used to ju…

Peer review is broken not because the process of review is broken but because the peers are broken. Reviewers, like all academics, have screwed up incentives to get there name on as many papers as possible and those papers cited widely, regardless of the quality of the science involved. They will always find a way game the system to their benefit. Recently heard about reviewers of a friend's paper blocking publicatio…

I'm only shocked that this is the first time you'd heard of adversarial delaying. When that fails, sometimes PIs will call up their editorially inclined pals and suggest a "bundle" issue to sneak in their lab's work. There are many ways to avoid ceding priority for discovery.

If preprint deposition is the default, this all goes away. The people who lose from such a shift are editors of major journals, publishers of journals, and senior PIs who know how to rig the game. A cynical mind might propose that these are the major forces resisting such a shift.

Re: Let’s make peer review scientific

#109

Post by apathy » Wed, Jul 06, 2016, 10:14 PM UTC

jsprogrammer wrote: post #73

lordnacho wrote: post #3

Perhaps it's a question of incentives. What exactly do you get out of reading a quite complex piece of work and saying your opinion? The closest I can come, as a non-academic, is perhaps reading other people's code. It's bloody hard. You need to concentrate, and it's not like reading a newspaper article at all. Even small errors are not easy to spot, and it's even hard to know whether the code is structured in the wa…

Reviews shouldn't be opinion, but facts. Those facts can then be disseminated to larger audiences. The incentive is to ensure that only factual information is claimed in publication so that everyone else can be as informed as possible, thereby (hopefully) allowing us to all make better decisions (and, perhaps, this could even accelerate the above described process).

Every time you use the word "should" you beg the question.

Why, exactly, are incentives aligned to make it so? Or is the problem that the incentives are not in fact set up to encourage the desired outcome?

Anything else is wishful thinking. People seldom do what they know to be right; they do what is convenient, then repent. It follows that one must make the right thing the convenient or rewarding thing if one wishes to see humans doing it.

By and large, people are somewhat predictable. This can be used for good as well as the more familiar evil uses.

Re: Let’s make peer review scientific

#110

Post by apathy » Wed, Jul 06, 2016, 10:21 PM UTC

yayday wrote: post #67

randall wrote: post #14

Earlier quoted context omitted.

But you don't need the 'gatekeeper'. Imagine if you had a 'trustability' factor that people could cryptographically sign as themselves. You could build a 'trust web', where papers link to other papers by an author. That author's trust ratings could go up based on the citer's evidence. There's clearly tech waiting to be built here, which might or might not be a startup.

There's already a "trustability" factor: it's called a name. When I search for a new paper and find something interesting, I look at the first author, the name of the PI, and the name of the university or laboratory associated with this paper. If I'm familiar with the work of the first author then I already have an idea of how much I can trust his/her work. If I don't know this person then I look at the coauthors and…

You don't consider the journal? You're a better man (or woman) than I am, if so. Otherwise agreed, reluctantly. This implicit trust is frequently manipulated to relax standards by experienced PIs, and in biomedical work that is a source of numerous problems.

In an ideal world, work from an unknown author would receive the same scrutiny as that from an established lab, and vice versa. Obviously this isn't the case, but the further we drift from it, the more likely we become to see sloppy results wasting everyone's time.

It can't just be about the narrative. That simply isn't science. That's storytelling, and confusing the two has caused a great deal of harm to science in the public eye.