Live data from Hacker News

Let’s make peer review scientific

nature.com

91–100 of 146 posts

Re: Let’s make peer review scientific

#91

Earlier quoted context omitted.

How are you planning on addressing the fact that none of this matters in the slightest if academic career growth/incentives/reputation specifically revolves around publishing boring pdfs in established journals? What do I gain by sending my work to your platform where the first question anyone in my "target audience" will ask is "why didn't this get published in a real journal, whats wrong with it?", and I will get e…

In mathematics, Perelman posted his beautiful 3 part proof to Poincare to arXiv and within a day there was allegedly hype (or so reputed professors report, while I did go into advanced alg. top. later, I was 14 at the time). Professors (the same professors who'd referee) were independently reviewing Part 1 within 24 hours and this was in 2002. Then you have a Fields Medal winner Voevodsky who also bitches about havin…

My god I love coming here and reading the travails of real scientists. The journals that have a "no preprint" policy are primarily Cell Press (link) rags and Annals of Oncology.

http://crosstalk.cell.com/blog/lets-talk-about-preprint-serv...

Note the insane amount of FUD. "We're fine with preprints, as long as we can control them!" But that's kind of the point, isn't it -- wrestling back control from journals with a massive inherent COI! Allowing the public (lay or scientific) to objectively determine what's real and what isn't, with visible salvos from both camps as needed, instead of the invisible and un-accountable hand of an editor or anonymous referees? (I sign my reviews.)

Physics doesn't seem to have suffered from preprints -- if anything, it has thrived more than expected thanks to it. I don't hear about a replicability crisis in physics, for some reason. I hear about it all the time in biomedical sciences, perhaps because the review process is so incredibly shady for the "top" journals (in terms of impact factor, a statistically indefensible and criminally manipulated metric). Papers that are outright rejected or in need of major revisions are simply yanked and resubmitted elsewhere (I am guilty of this, by the way; rejected from Nature, so we sent it to NEJM with almost no edits and they accepted it a week later). Senior authors act rationally, so if perfecting a paper will cause a delay and potentially deny them priority on a discovery, who can argue for making the edits instead of resubmitting elsewhere?

All of this goes away when preprints are the default. Priority is established by deposition, rather than publication; the discovery often (though perhaps not always) is itself enough to prompt review from rivals, even if only in self-interest; and the preprint status makes it clear that the item has not been formally reviewed. Many or most resulting referee comments are public, often submitted as related deposits. From a scholarship perspective, I cannot for the life of me see the downside, other than reticence to boldly state the truth and face retaliation from shysters. It's a shame that the latter is an issue, but given the even larger amounts of $$$ at stake in physics, it seems that biomedical researchers are mostly just cowards using this as an excuse. (JMHO)

Re: Let’s make peer review scientific

#92
post #22
post #8

I'm enclined to think that an open review process by using a web system with a space for open discussion/commenting could be a good system. This wasn't possible before the web. Actual system is publish or not, but it is possible now to publish everything and use a rating system. This removes the risk of plagiarism and anteriority disputes.

Isn't 'the scientific paper' a kind of dated concept? I'd love to see the whole scientific process from start to finish in a 'notebook' form. Why summarize findings in a paper at the end of it all, when all steps (including conception, data gathering, successes, missteps, comments, and corrections) could all be viewed as a whole? It would seem to be more in line with the intent of science. I'm not a scientist, obviou…

Because a summary is very useful.

Imagine you are reading about 5 years of day-in-day-out work by a team of 1500 physicists (eg. CERN, LHC, etc...).

Would you rather read the entire as-it-happened stream of research, going back 7500 person-years - or would you prefer to read a summary, in the form of a paper?

I agree that ideally the stream should be publically available as well, for those that want to consult it - but having regular summaries, written at intervals is very useful.

Re: Let’s make peer review scientific

#93
post #20

Shameless plug: I'm working at the MIT Media Lab on the PubPub project ( http://www.pubpub.org ), a free platform for totally open publishing designed to solve a lot of these problems: One is peer review, which, as some have already mentioned, needs to be done in an open, ongoing, and interactive forum. Making peer review transparent to both parties (and the public) makes everyone more honest. Another is the incentiv…

I still don't... get what PubPub is. I definitely think that peer review should be updated and science should be using the internet better, I just have no idea what PubPub has to do with that mission. Is it just... a place where people can post things, and other people can comment, and other people can curate posts? So like any blogging platform?

Even if used as intended, it seems like the only comments would go on things that people are already seeking out; there hasn't been a "random pub" button since it got overtaken with spam. So the use of comments to "review" is already throwing up a huge barrier to discovering new science and leveling the playing field for unrecognized researchers, exacerbating the problems of the current system. The inability to anonymize comments or posts gets rid of the most promising way the internet could affect peer review. It's hard to see what about the platform promotes evolving documents as opposed to static publications. And complex, digitally native papers are already being promoted by larger journals, with better resources to implement them, and more staff to help potentially unfamiliar scientists with the process, another aspect that could make the platform a source of exclusion and bias.

I don't mean to sound to critical, I just follow both science publishing and projects coming out of the Media Lab pretty closely, and have been baffled by this one for a while. https://www.sciencematters.io/ is doing cool things with switching from static publications to a set of growing datapoints, updated in real time, http://meta.com/ (formerly sciencescape) is dipping their toes into curating papers in a really interesting way, and even http://the-artifice.com/ has a unique peer review system that I think could be implemented well in the hard sciences as well. PubPub is just like... Medium? With a little bit of GitHub, and the ethos of arXiv?

Re: Let’s make peer review scientific

#94
post #82
post #79

Earlier quoted context omitted.

> Maybe we could call this "depository" of money to conduct research something like, hmmm, what's a good word… a grant? Nah, that word is already in use for stagnant allocations of academic welfare to work on bullshit instead of transformative techniques (e.g. CAR T-cells, which NIH refused to fund for years). Need a new word to signify "money that is actually intended to produce results" instead of "a pension for ir…

> or NSF (whose yearly funding is dwarfed by the R&D funding at a single company) If the worst thing you can say about the NSF is that they need more money, that makes it sound like GP has come up with a nice way to allocate the available funding towards particular research projects. > It would be nice if you thought a little more carefully about what you wrote. The devil is in the details. Details like how to get "c…

I take it you're not familiar with "crowdfunding" sources like the AACR, LLS, ASCO, or other professional societies?

As someone who is funded by several of the above, and who noted that their review processes were substantially less bullshit-intensive yet no less rigorous than NIH review (which has many benefits, efficiency not among them), I'm going to go out on a limb and suggest that it's possible.

As far as the NSF, they do a good job with what they have, but what they have is not commensurate with what we as a society could stand to spent on science. Even NCI is a far cry from that: https://pbs.twimg.com/media/CmLJzKQWkAAl372.jpg:small

Distributions are similar for various other avenues of funding, and it is quite clear that the overhead & administrative costs requested by many recipient instutions are far out of proportion to actual needs, so the impact of the funding allocations is further reduced.

Thus it appears that a direct conduit from potential patrons to researchers is, in fact, desirable. Otherwise, services like experiment.com would not exist. They're not at the level of an NIH study section (duh?) but they have consistently produced a small stream of usable results that belie their supposed irrelevance. Once upon a time, the Royal Society existed for just such matchmaking: find a rich patron and a promising young scientist and line them up. You've likely noticed that many if not most major universities and "centers of excellence" rely upon exactly this model, supplemented with NIH or NSF grants, to exist. Further modularizing the model so that an administrative hand yanking out bloated "indirects" at every turn is not mandated might not be the worst thing, or (alternatively) being more transparent with said O&A requests, might at least bring some of the bullshit under control.

The public clearly wants accountability. The masses may be asses, but if we want their money, we really ought to be transparent about what we're doing with it.

Re: Let’s make peer review scientific

#95
post #82
post #79

Earlier quoted context omitted.

> Maybe we could call this "depository" of money to conduct research something like, hmmm, what's a good word… a grant? Nah, that word is already in use for stagnant allocations of academic welfare to work on bullshit instead of transformative techniques (e.g. CAR T-cells, which NIH refused to fund for years). Need a new word to signify "money that is actually intended to produce results" instead of "a pension for ir…

> or NSF (whose yearly funding is dwarfed by the R&D funding at a single company) If the worst thing you can say about the NSF is that they need more money, that makes it sound like GP has come up with a nice way to allocate the available funding towards particular research projects. > It would be nice if you thought a little more carefully about what you wrote. The devil is in the details. Details like how to get "c…

I feel like those fields with highest facility needs / costs would come last if at all. There are many fields that require pretty small amounts of resources for example: Computer Science (I did most of my research on a personal laptop, with other equipment costs All of these seem very possible to crowdfund with the ultimate goal of unhooking them from perverse incentive systems of typical universities.

Re: Let’s make peer review scientific

#96
post #82
post #79

Earlier quoted context omitted.

> Maybe we could call this "depository" of money to conduct research something like, hmmm, what's a good word… a grant? Nah, that word is already in use for stagnant allocations of academic welfare to work on bullshit instead of transformative techniques (e.g. CAR T-cells, which NIH refused to fund for years). Need a new word to signify "money that is actually intended to produce results" instead of "a pension for ir…

> or NSF (whose yearly funding is dwarfed by the R&D funding at a single company) If the worst thing you can say about the NSF is that they need more money, that makes it sound like GP has come up with a nice way to allocate the available funding towards particular research projects. > It would be nice if you thought a little more carefully about what you wrote. The devil is in the details. Details like how to get "c…

Also: I still haven't heard (from either you or the previous parent poster) how journal impact factor can possibly be justifiable as a metric for relevance.

Anyone surveying the actual citation distributions at major journals will immediately note that a metric assuming near-normality cannot possibly summarize non-normal distributions of citations. The latter describes nearly all journals, thus even if JIF were not manipulable by stacking, self-citation, and negotiated exclusion of items to decrease the denominator, it would still suck.

https://quantixed.wordpress.com/2016/01/05/the-great-curve-i...

Look carefully at the details! This metric is among the most frequently emphasized by researchers who comprise study sections, and it is objectively terrible.

I'm not whining "just because" -- many of the lines in my CV end with NEJM, Nature, or Cell (no Science paper yet). I'm saying that at least one of the commonly accepted metrics for individual investigators is broken. That sort of detail corrupts the entire rest of the system.

I'm also not saying that a direct public-facing system wouldn't have huge potential problems (although it is nice to see attempts like experiment.com seemingly doing OK, and the funders realizing, hey, there are a lot of shades of gray between "utter bullshit" and "exactly the right experimental design for the question being asked").

One of the nice things about talking directly with folks at NIH, for example, is that they recognize there are serious issues with the incentives in place. If they are willing to collect the data and evaluate (publicly, e.g. in Chalk Talk postings) the findings, doesn't that suggest room for the current system to improve?

Re: Let’s make peer review scientific

#97

Earlier quoted context omitted.

I am in Australia where the pool of potential grant reviewers is pretty small for most areas. When you have a grant success rate of 15% it is all too easy to nobble your competitors via the review process.

Why is science viewed as a competition there?

Because there's a finite pool of money.

Re: Let’s make peer review scientific

#98
post #76

Earlier quoted context omitted.

> where the first question anyone in my "target audience" will ask is "why didn't this get published in a real journal, whats wrong with it?" One would think that this wouldn't happen simply because it is (I think) a combination of two of the most well-known logical fallacies, "argumentum ad populum" and "appeal to authority" https://yourlogicalfallacyis.com/appeal-to-authority I am extremely dismayed by the fact tha…

Conferences and journals are a filter. If your paper can't pass a few rounds of anonymized review (for a conference) then it probably isn't worth the readers time. As an example, arXiv is has a lot of great papers but it also has a lot of bad ones. I'm unlikely to go there to find a paper unless someone referred me to a specific one.

I've never had a paper rejected from a conference, and I've seen some pretty shitty ones at conferences. I think this statement is pretty field-specific.

That said, in the CS and EE fields at least, the use of 10-12 referees for a short, dense conference paper is a really good idea. It would appear that this works primarily because of... drum roll ...the social norms and incentives in place within these specific fields.

Re: Let’s make peer review scientific

#99

Earlier quoted context omitted.

Why reinvent this wheel? Why not instead (also) partner with people like Overleaf who already have many of these features? https://www.overleaf.com

that appears to be totally proprietary

not really

https://github.com/overleaf

Some parts of it are (duh?) but overall the system is astoundingly transparent. I think they ended up charging for the sync-to-git option, but if you look at the above repos, it appears that this feature can actually be pulled on the user end rather than just pushed from Overleaf.

Overleaf integrates nicely with existing journals and with preprint servers to some extent; I've bugged them for some time to support MarkDown (I don't care to use LaTeX for manuscripts which don't have much math) but as yet they have not implemented it. It may be an issue with the backend, although even that should be surmountable.

Re: Let’s make peer review scientific

#100
post #82

Earlier quoted context omitted.

> or NSF (whose yearly funding is dwarfed by the R&D funding at a single company) If the worst thing you can say about the NSF is that they need more money, that makes it sound like GP has come up with a nice way to allocate the available funding towards particular research projects. > It would be nice if you thought a little more carefully about what you wrote. The devil is in the details. Details like how to get "c…

I feel like those fields with highest facility needs / costs would come last if at all. There are many fields that require pretty small amounts of resources for example: Computer Science (I did most of my research on a personal laptop, with other equipment costs All of these seem very possible to crowdfund with the ultimate goal of unhooking them from perverse incentive systems of typical universities.

at least three of the above are in fact supported by experiment.com backers, although largely as a "bridge" to more traditional scholarly outlets. That said, if you go out and get extramural funding for your work, generally that is the definining characteristic of a successful PI, so...
Post reply on HN