Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let’s make peer review scientific

nature.com

Post reply on HN
Print view View original

71–80 of 146 posts

Re: Let’s make peer review scientific

#71

Post by djokkataja »

Jabbles wrote: post #19
mariusz79 wrote: post #16

I've got a simpler solution - scientific paper should not be considered valid until another team replicates the findings. That would quickly get rid of all of the fake results, plus would weed out all of the research that nobody really cares about. Also require that all data is shared, and that until your results are not verified or replicated by a team from another university/country you don't get any more funds.

That's not going to work for Astronomy, or results of multi-year studies.

Long-term studies or not, it would be useful to have an up-to-date system that clearly indicated whether or not the results of some particular study had been replicated, and how many times they had been replicated if at all, and if there were any conflicting results between studies.

Re: Let’s make peer review scientific

#72

Post by jsprogrammer »

danieltillett wrote: post #30
dangerlibrary wrote: post #12

Earlier quoted context omitted.

At the NIH, at least, I know that enormous effort is put into ensuring that there are not conflicts of interest between the reviewers and those being reviewed. I'm not particularly surprised to hear that some places do not put in that effort - it is a slow, painstaking process - but there are places where this is less of a problem.

I am in Australia where the pool of potential grant reviewers is pretty small for most areas. When you have a grant success rate of 15% it is all too easy to nobble your competitors via the review process.

Why is science viewed as a competition there?

Re: Let’s make peer review scientific

#73

Post by jsprogrammer »

lordnacho wrote: post #3

Perhaps it's a question of incentives. What exactly do you get out of reading a quite complex piece of work and saying your opinion? The closest I can come, as a non-academic, is perhaps reading other people's code. It's bloody hard. You need to concentrate, and it's not like reading a newspaper article at all. Even small errors are not easy to spot, and it's even hard to know whether the code is structured in the wa…

Reviews shouldn't be opinion, but facts. Those facts can then be disseminated to larger audiences. The incentive is to ensure that only factual information is claimed in publication so that everyone else can be as informed as possible, thereby (hopefully) allowing us to all make better decisions (and, perhaps, this could even accelerate the above described process).

Re: Let’s make peer review scientific

#74

Post by pmarreck »

suchsciencewow wrote: post #41
joelg wrote: post #20

Shameless plug: I'm working at the MIT Media Lab on the PubPub project ( http://www.pubpub.org ), a free platform for totally open publishing designed to solve a lot of these problems: One is peer review, which, as some have already mentioned, needs to be done in an open, ongoing, and interactive forum. Making peer review transparent to both parties (and the public) makes everyone more honest. Another is the incentiv…

How are you planning on addressing the fact that none of this matters in the slightest if academic career growth/incentives/reputation specifically revolves around publishing boring pdfs in established journals? What do I gain by sending my work to your platform where the first question anyone in my "target audience" will ask is "why didn't this get published in a real journal, whats wrong with it?", and I will get e…

> where the first question anyone in my "target audience" will ask is "why didn't this get published in a real journal, whats wrong with it?"

One would think that this wouldn't happen simply because it is (I think) a combination of two of the most well-known logical fallacies, "argumentum ad populum" and "appeal to authority" https://yourlogicalfallacyis.com/appeal-to-authority

I am extremely dismayed by the fact that, in a career where (of ALL careers!) the evidence should ostensibly speak for itself, papers (and their information) are essentially pre-rejected or pre-accepted based on reputation

Re: Let’s make peer review scientific

#75

Post by yayday »

zenogais wrote: post #70
isTravis wrote: post #44

Earlier quoted context omitted.

Great question, and one that certainly doesn't have a straightforward or trivial answer. It's definitely more of a social challenge than a technical one - making publishing free/open won't do anything to fix incentives on its own. My hunch is that change to this system will come from the outside. It's too risky of a career decision for a tenure-track professor to start publishing on PubPub (or any open/new system). B…

Perhaps offering some sort of crowdsourced funding mechanism and a reputation system would go a long way toward correcting some of these incentives? For example, giving authors / organizations a Bitcoin address where they can receive funds from individuals / organizations who want to support their research. Also, awarding reputation to authors based on the level of peer review their research has successfully undergon…

> For example, giving authors / organizations a Bitcoin address where they can receive funds from individuals / organizations who want to support their research.

That's a fantastic idea. Maybe we could call this "depository" of money to conduct research something like, hmmm, what's a good word… a grant?

> Also, awarding reputation to authors based on the level of peer review their research has successfully undergone (number of peers, level of rigor, etc.), and conversely awarding reputation and funding to those who perform peer reviews.

Sounds fantastic as well! Maybe these authors could create like, a website or curriculum vitae where they could list their accomplishment to establish their reputation. You know, they could have a section in their medium of choice that could be titled something like selected peer reviewed articles where they'll list their publications along with their coauthors and the journal it appeared in. Maybe these journals could devise some kind of ranking to measure reputation. Maybe they could call it something like… amount of impact or maybe just impact factor for short. I think this could work really well.

> Allowing users to contribute to a peer review fund for individual articles or in general.

Maybe a general fund should be created to support science! Maybe a national science fund or something, governed by a so-called national science foundation who can vote scientists, engineers, and the like onto their board to steer the allocation of funding.

I really think you're onto something very good here!

Re: Let’s make peer review scientific

#76

Post by zerohp »

pmarreck wrote: post #74
suchsciencewow wrote: post #41

Earlier quoted context omitted.

How are you planning on addressing the fact that none of this matters in the slightest if academic career growth/incentives/reputation specifically revolves around publishing boring pdfs in established journals? What do I gain by sending my work to your platform where the first question anyone in my "target audience" will ask is "why didn't this get published in a real journal, whats wrong with it?", and I will get e…

> where the first question anyone in my "target audience" will ask is "why didn't this get published in a real journal, whats wrong with it?" One would think that this wouldn't happen simply because it is (I think) a combination of two of the most well-known logical fallacies, "argumentum ad populum" and "appeal to authority" https://yourlogicalfallacyis.com/appeal-to-authority I am extremely dismayed by the fact tha…

Conferences and journals are a filter. If your paper can't pass a few rounds of anonymized review (for a conference) then it probably isn't worth the readers time.

As an example, arXiv is has a lot of great papers but it also has a lot of bad ones. I'm unlikely to go there to find a paper unless someone referred me to a specific one.

Re: Let’s make peer review scientific

#78

Post by zenogais »

yayday wrote: post #75
zenogais wrote: post #70

Earlier quoted context omitted.

Perhaps offering some sort of crowdsourced funding mechanism and a reputation system would go a long way toward correcting some of these incentives? For example, giving authors / organizations a Bitcoin address where they can receive funds from individuals / organizations who want to support their research. Also, awarding reputation to authors based on the level of peer review their research has successfully undergon…

> For example, giving authors / organizations a Bitcoin address where they can receive funds from individuals / organizations who want to support their research. That's a fantastic idea. Maybe we could call this "depository" of money to conduct research something like, hmmm, what's a good word… a grant ? > Also, awarding reputation to authors based on the level of peer review their research has successfully undergone…

Oooo sarcasm. You're probably right though, the old system seems to be working out pretty well. Besides, science is all about never questioning existing institutions right?

Re: Let’s make peer review scientific

#79

Post by apathy »

yayday wrote: post #75
zenogais wrote: post #70

Earlier quoted context omitted.

Perhaps offering some sort of crowdsourced funding mechanism and a reputation system would go a long way toward correcting some of these incentives? For example, giving authors / organizations a Bitcoin address where they can receive funds from individuals / organizations who want to support their research. Also, awarding reputation to authors based on the level of peer review their research has successfully undergon…

> For example, giving authors / organizations a Bitcoin address where they can receive funds from individuals / organizations who want to support their research. That's a fantastic idea. Maybe we could call this "depository" of money to conduct research something like, hmmm, what's a good word… a grant ? > Also, awarding reputation to authors based on the level of peer review their research has successfully undergone…

> Maybe we could call this "depository" of money to conduct research something like, hmmm, what's a good word… a grant?

Nah, that word is already in use for stagnant allocations of academic welfare to work on bullshit instead of transformative techniques (e.g. CAR T-cells, which NIH refused to fund for years). Need a new word to signify "money that is actually intended to produce results" instead of "a pension for irrelevant tenured senior PIs to pay non-English-speakers below-minimum-wage to work on topics that became irrelevant a decade ago".

> Maybe they could call it something like… amount of impact or maybe just impact factor for short. I think this could work really well.

Ah yes, impact factor is such an amazing tool. It allows "executive" "leadership" types to predict (very poorly, but who cares?) how many citations a paper might receive if it survives the months or years between submission and publication in a major journal. Trouble is, JIF is massively massaged and the COI that Thompson Reuters has in equitably enforcing it is ridiculous.

WARNING: Non-peer-reviewed work ahead! If you're not careful, you might have to apply critical thinking to it!

http://biorxiv.org/content/early/2016/07/05/062109

> Maybe a general fund should be created to support science!

That's a great theory. Perhaps it can be as well executed as the CIHR fund (where study section has given way to "ignore everyone who doesn't suck my dick directly") or NSF (whose yearly funding is dwarfed by the R&D funding at a single company). This approach is working out very well!

You know, if I didn't know better, I might think you were the sort of researcher that fails to look at the details and just submits your most fashionable bullshit to whateve journal at which your pals happen to be editors. I might get the impression that you're the cancer which is killing grant-funded science, which prizes large labs over large numbers of R01 projects, which believes that O&A is an entitlement to take out mortgages on new buildings instead of to pay for the costs of disbursing and administering a grant. But, since the evidence isn't before me, I won't.

It would be nice if you thought a little more carefully about what you wrote. The devil is in the details.

Re: Let’s make peer review scientific

#80

Post by apathy »

Buried in the middle of this (wonderful) article is the heart of the problem -- academia has foolishly placed its metrics into the hands of editors and publishers, who have corrupted the living hell out of it. Ctrl-F "Cochrane" and witness the exchange between an editor, who benefited from the status quo, and a scholar, who did not and does not.

Academia != scholarship and has not for some time. There is no longer a good reason for traditional for-profit journals to exist. (Before someone says it: SpringerNature likes to pretend that editorial independence is possible, but they'll have no choice save to fire their "news" guys if their board asks for this).

Please recall that the entire point of the World Wide Web was to share physics papers. The arXiv exists because of physicists (who quickly noticed that HTML wasn't a good substitute for LaTeX when writing math-heavy papers). The problem is not technological. It's social. And until the incentives are fixed (the Cochrane Collaboration in Britain has gone a long, long way to address this, and now the Wellcome Trust is going even further), nothing of any real import will change. In the USA, NIH could make a lot of positive changes (and in exchanges with mid- to senior executive level directors, I honestly believe they're trying to do so). But it will take time. Academia moves with glacial speed, when it moves at all.

Post reply on HN