Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let’s make peer review scientific

nature.com

Post reply on HN
Print view View original

121–130 of 146 posts

Re: Let’s make peer review scientific

#121

Post by jsprogrammer »

apathy wrote: post #109
jsprogrammer wrote: post #73

Earlier quoted context omitted.

Reviews shouldn't be opinion, but facts. Those facts can then be disseminated to larger audiences. The incentive is to ensure that only factual information is claimed in publication so that everyone else can be as informed as possible, thereby (hopefully) allowing us to all make better decisions (and, perhaps, this could even accelerate the above described process).

Every time you use the word "should" you beg the question. Why, exactly, are incentives aligned to make it so? Or is the problem that the incentives are not in fact set up to encourage the desired outcome? Anything else is wishful thinking. People seldom do what they know to be right; they do what is convenient, then repent. It follows that one must make the right thing the convenient or rewarding thing if one wishes…

Reviews should be facts by definition of a review (as I use it), so the question is not begged.

The incentives are 'aligned' that way because it is the only way to benefit. Publishing into an echo chamber (or void) is not science, by definition.

Re: Let’s make peer review scientific

#122

Post by yayday »

zenogais wrote: post #78
yayday wrote: post #75

Earlier quoted context omitted.

> For example, giving authors / organizations a Bitcoin address where they can receive funds from individuals / organizations who want to support their research. That's a fantastic idea. Maybe we could call this "depository" of money to conduct research something like, hmmm, what's a good word… a grant ? > Also, awarding reputation to authors based on the level of peer review their research has successfully undergone…

Oooo sarcasm. You're probably right though, the old system seems to be working out pretty well. Besides, science is all about never questioning existing institutions right?

It was sarcasm yes but my point was that what you proposed already exists, just without the crypto currency bullsht and extra hoops to jump through.

I didn't say the current system was flawless. I'm just saying your proposal is the current reality already.

Re: Let’s make peer review scientific

#123

Post by yayday »

apathy wrote: post #79
yayday wrote: post #75

Earlier quoted context omitted.

> For example, giving authors / organizations a Bitcoin address where they can receive funds from individuals / organizations who want to support their research. That's a fantastic idea. Maybe we could call this "depository" of money to conduct research something like, hmmm, what's a good word… a grant ? > Also, awarding reputation to authors based on the level of peer review their research has successfully undergone…

> Maybe we could call this "depository" of money to conduct research something like, hmmm, what's a good word… a grant? Nah, that word is already in use for stagnant allocations of academic welfare to work on bullshit instead of transformative techniques (e.g. CAR T-cells, which NIH refused to fund for years). Need a new word to signify "money that is actually intended to produce results" instead of "a pension for ir…

I wasn't saying the current system is great, there's a lot wrong with it yes I agree. The impact factor thing is also a pretty silly metric to me as well, I agree with you there. The point I was trying to make, albeit sarcastically, was that the system the guy proposed is what we have today just without the extra hoops to jump through. Like I have absolutely no interest in maintaining a Bitcoin wallet or whatever nor do I want anything to do with them. I'll take my funding in dollars or euros or something real and tangible please.

Re: Let’s make peer review scientific

#124

Post by yayday »

jsprogrammer wrote: post #101
chmike wrote: post #56

Earlier quoted context omitted.

arxiv is not open. At least not as open as it should be in my opinion. But I fully agree on the chicken-or-egg problem. The main problem is bootsrapping such web service. I don't think that "anoiting" a forum/web service would be a good strategy. We need to keep open the possibility of a new and better system to take over. My feeling is that two fundamental requirements must be satisfied. 1. Anybody should be able to…

One could start with a model similar to that in which most people using systems which this new system is meant to replace already find themselves. You should be able to select peers (though, maybe you should be given some as well?) and "experts" to follow. You would only see their activity when you want to, but you could also participate in the larger network. As a practical matter of implementation, I think allowing…

You just described LinkedIn. A website people only update every few months/years when they're looking for a new job.

Re: Let’s make peer review scientific

#125

Post by yayday »

apathy wrote: post #110
yayday wrote: post #67

Earlier quoted context omitted.

There's already a "trustability" factor: it's called a name. When I search for a new paper and find something interesting, I look at the first author, the name of the PI, and the name of the university or laboratory associated with this paper. If I'm familiar with the work of the first author then I already have an idea of how much I can trust his/her work. If I don't know this person then I look at the coauthors and…

You don't consider the journal? You're a better man (or woman) than I am, if so. Otherwise agreed, reluctantly. This implicit trust is frequently manipulated to relax standards by experienced PIs, and in biomedical work that is a source of numerous problems. In an ideal world, work from an unknown author would receive the same scrutiny as that from an established lab, and vice versa. Obviously this isn't the case, bu…

I'd be lying if I said the journal didn't matter strictly speaking, but in general as long as it's a "real" journal (as opposed to one of those obviously fake journals I keep getting spammed by) then I don't really care.

Don't get me wrong though, just because the paper has a heavy hitter as the PI or even as first author, doesn't mean I treat the content with less scrutiny. It's more that I know my time is extremely limited so I'm more willing to spend more time dissecting a paper from an established first author or PI than an unknown person (unknown also meaning that it's not a paper someone recommended to me or a paper someone cited in another work).

It's more like, I'll _make_ time to read papers with certain authors on it because I know that they do good work. I'll also _make_ time to read something if it's published in Nature et al for example. But a random paper from a random author in a random group? It's not that I don't trust this person's work, it's more that I'm unfamiliar with that person's work and there's other things I could spend my time doing as well. I'll download the paper, throw it on my iPad, etc, and I'll get around to it eventually.

Re: Let’s make peer review scientific

#126

Post by kaybe »

semi-extrinsic wrote: post #106
joelg wrote: post #20

Shameless plug: I'm working at the MIT Media Lab on the PubPub project ( http://www.pubpub.org ), a free platform for totally open publishing designed to solve a lot of these problems: One is peer review, which, as some have already mentioned, needs to be done in an open, ongoing, and interactive forum. Making peer review transparent to both parties (and the public) makes everyone more honest. Another is the incentiv…

> Another is the nature of the medium itself. It's 2016, and these dead, flat, static PDFs are functionally identical to the paper they replaced! Well, many of the existing publishers (at least ACS, Nature group, Wiley, Science group) are already pushing hard towards something "more" than PDF, mainly for DRM reasons. And scientists in general are resisting greatly, because PDF is such an established format, supported…

I'm also concerned about long term readability. Some of those documents might have at least a few hundred years of people wanting to read them before them. Can any of those innovative new data formats stay around long enough, especially if they're proprietary? We might change computation platform and hardware paradigms a few times as well, just think back to 20 years ago. A lot of measured raw data is already lost due to this; it's about impossible to parse any more. Pure text can survive far more easily.

Re: Let’s make peer review scientific

#127

Post by untilHellbanned »

apathy wrote: post #105
untilHellbanned wrote: post #52

Earlier quoted context omitted.

As an academic in biomedical sciences and as someone who tried to create a journal that did all the things the OP said, thank you. The old crappy system is mandatory. You can't decide to do otherwise and still have a job. The impetus for change needs to start at the funding source level, NIH. They need to make a journal that one is required to publish in order to keep your funding.

They have one (several, actually). The trouble is that they're called Cell, Nature, and Science; they're all for-profit; and they answer to no one. The Wellcome Trust is walking the walk that NIH talked for the past decade. We will see how this goes. Cross fingers.

With ELife?

Re: Let’s make peer review scientific

#128

Post by quadrangle »

apathy wrote: post #99
quadrangle wrote: post #57

Earlier quoted context omitted.

that appears to be totally proprietary

not really https://github.com/overleaf Some parts of it are (duh?) but overall the system is astoundingly transparent. I think they ended up charging for the sync-to-git option, but if you look at the above repos, it appears that this feature can actually be pulled on the user end rather than just pushed from Overleaf. Overleaf integrates nicely with existing journals and with preprint servers to some extent; I've bu…

not sure how to parse all that. At the least, it's not straightforward, and my original comment is 100% true: "that appears to be totally proprietary" (emphasis added)

If a project is actually FLO, you'd think they'd make that clear to someone visiting their website.

Re: Let’s make peer review scientific

#129

Post by jsprogrammer »

yayday wrote: post #124
jsprogrammer wrote: post #101

Earlier quoted context omitted.

One could start with a model similar to that in which most people using systems which this new system is meant to replace already find themselves. You should be able to select peers (though, maybe you should be given some as well?) and "experts" to follow. You would only see their activity when you want to, but you could also participate in the larger network. As a practical matter of implementation, I think allowing…

You just described LinkedIn. A website people only update every few months/years when they're looking for a new job.

The content focus is different, at least.

In addition to collaborative document authoring (which I don't believe LinkedIn provides), data, simulations, and analysis code could be shared and built-on.

Re: Let’s make peer review scientific

#130

Post by jsprogrammer »

dflock wrote: post #89
jsprogrammer wrote: post #72

Earlier quoted context omitted.

Why is science viewed as a competition there?

For exactly the same reasons it is everywhere in the world: because there isn't enough funding to go around - and the grant application process is explicitly designed to be competitive.

It doesn't seem to be everywhere though. I know in the US, research is often collaborative and a single grant may fund many researchers. Those researchers do not seem to be competing internally for funding (though, I have seen conflict as to whose name should appear first on a publication!). I presume the same is likely true in Australia. Why is some research able to be collaborative while other must (apparently) be competitive?
Post reply on HN