Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Replace peer review with “peer replication” (2021)

blog.everydayscientist.com

Post reply on HN
Print view View original

221–230 of 351 posts

Re: Replace peer review with “peer replication” (2021)

#221

Post by dmbche »

mike_hearn wrote: post #191
matthewdgreen wrote: post #60

Earlier quoted context omitted.

No subset of humanity is “pure hearted.” Fraud and malice will exist in everything people do. Fortunately these fraudulent incidents seem relatively rare, when one compares the number of reported incidents to the number of publications and scientists. But this doesn’t change anything. The benefit of scientific publication is to make it easier to detect and verify incorrect results , which is exactly what happened in…

Fraud isn't exceedingly rare :( It only seems that way because academia doesn't pay anyone to find it, reacts to volunteer reports by ignoring it, and the media generally isn't interested. Fraud is so frequent and easy to find that there are volunteers who in their spare time manage to routinely uncover not just individual instances of fraud but entire companies whose sole purpose is to generate and sell fake papers…

Thanks you - just discovered Scigen, these links are incredible

Re: Replace peer review with “peer replication” (2021)

#222

Post by tnecniv »

vladms wrote: post #136
lliamander wrote: post #117

Earlier quoted context omitted.

I agree that most labs are probably not out to defraud people. But without replication I don't think it's reasonable to have much confidence in what is published.

Maybe replication should (and probably does) happen when the published thing is relevant to some entity and also interesting. I never seen papers as "truth", but more as "possibilities". After many other "proofs" (products, papers, demos, etc.) you can assign some concepts/ideas the label "truth" but one/two papers from the same group is definitely not enough.

Yeah passing peer review doesn’t mean that the article is perfect and to be taken as truth now (and remember, to err is human; any coder on here has had some long standing bug that went mostly unnoticed in their code base). It means it passed the journal’s standards for novelty, interest, and rigor based on the described methods as a retained by the editor / area chair and peer reviewers that are selected for being knowledgeable on the topic.

Implicit in this process is that the authors are acting in good faith. To treat the authors as hostile is both demoralizing for the reviewers (who wants to be that cynical about their field) and would require extensive verification of each statement well beyond what is required to return the review in a timely manner.

Unless your paper has mathematical theory (and mistakes do slip through), a publication should not be taken as proof of something on its own, but a data point. Over time and with enough data points, a field builds evidence to turn a hypothesis into a scientific theory.

Re: Replace peer review with “peer replication” (2021)

#223

Post by ayakang31415 »

One of the Nobel prizes in Physics was the discovery of Higgs Boson at LHC. It cost billions of dollars just to build the facility, and required hundreds of physicists working on it to just conduct the experiment. You can't replicate this. Although I fully agree that replication must come first when it is reasonably doable.

Re: Replace peer review with “peer replication” (2021)

#224

Post by jononomo »

fabian2k wrote: post #15

I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…

If it is not replicated it shouldn't be published, other than as a provisional draft. I don't care if it hurts your feelings.

Re: Replace peer review with “peer replication” (2021)

#225

Post by johnnyworker »

MrJohz wrote: post #210
johnnyworker wrote: post #187

Earlier quoted context omitted.

> But the original result is useful in its own right - it tells us that there may be something worth exploring. I disagree. It shows that when someone writes something in a text editor and publishes it, others can read the words they wrote. That's all it shows, by itself. Just like someone writing something on the web only tells us that a textarea accepts just about any input. And even if it did show more than that,…

I think you may be seeing the purpose of these papers differently to me, which may be the cause of this confusion. The way you're describing a scientific publication is as if it were the end result of the scientific act. To use the software analogy, you're describing publication like a software release: all tests have been performed, all CI workflows have passed, QA have checked everything, and the result is about to…

> But talking to researchers, they see publishing more like making a new branch in a repository.

Well some do, others don't. Like the one who wrote the article this is a discussion of.

https://en.wikipedia.org/wiki/Replication_crisis

> Replication is one of the central issues in any empirical science. To confirm results or hypotheses by a repetition procedure is at the basis of any scientific conception. A replication experiment to demonstrate that the same findings can be obtained in any other place by any other researcher is conceived as an operationalization of objectivity. It is the proof that the experiment reflects knowledge that can be separated from the specific circumstances (such as time, place, or persons) under which it was gained.

Or, in short, "one is none". One might turn into more than one, it might not. Until it does, it's not real.

more snippets from the above WP article:

> This experiment was part of a series of three studies that had been widely cited throughout the years, was regularly taught in university courses

> what the community found particularly upsetting was that many of the flawed procedures and statistical tools used in Bem’s studies were part of common research practice in psychology.

> alarmingly low replication rates (11-20%) of landmark findings in preclinical oncological research

> A 2019 study in Scientific Data estimated with 95% confidence that of 1,989 articles on water resources and management published in 2017, study results might be reproduced for only 0.6% to 6.8%, even if each of these articles were to provide sufficient information that allowed for replication

I'm not saying it couldn't be fine to just publish things because they "could be interesting". But the overall situation seems like quite the dumpster fire to me. As does software, FWIW.

Re: Replace peer review with “peer replication” (2021)

#226

Post by majormajor »

tnecniv wrote: post #129
ebiester wrote: post #30

Earlier quoted context omitted.

It's simple but not easy: You create another path to tenure which is based on replication, or on equal terms as a part of a tenure package. (For example, x fewer papers but x number of replications, and you are expected to have x replications in your specialty.) You also create a grant funding section for replication which is then passed on to these independent systems. (You would have to have some sort of randomizat…

Your proposal has a whole slew of issues. First, people that want to be professors normally do so because they want to steer their research agenda, not repeat what other people are doing without contribution. Second, who works in their lab? Most of the people doing the leg work in a lab are PhD students, and, to graduate, they need to do something novel to write up in their dissertation. Thus, they can’t just replica…

> First, people that want to be professors normally do so because they want to steer their research agenda, not repeat what other people are doing without contribution.

If we're talking about weird incentives and academia you hit on one of the worst ones right here, I think, since nothing there is very closely connected to helping students learn.

I know that's a dead horse, but it's VERY easy to find reasons that we shouldn't be too closely attached to the status quo.

> For every lab doing new work, you’d basically need a clone of that lab to replicate their work.

Hell, that's how startup funding works, or market economies in general. Top-down, non-redundant systems are way more fragile than distributed ecosystems. If you don't have the competition and the complete disconnection, you so much more easily fall into political games of "how do we get this published even if it ain't great" vs "how do we find shit that will survive the competition"

Re: Replace peer review with “peer replication” (2021)

#227

Post by renonce »

vibrio wrote: post #156
kergonath wrote: post #50

Earlier quoted context omitted.

> I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. They also tend to over-estimate the effect of peer review (often equating peer review with validity). > If someone cares enough about the work to build on it, they will replicate it anyway. And in that case they have a good incentive to spend the effort…

“They also tend to over-estimate the effect of peer review (often equating peer review with validity).“ In my experience, scientists ate comfortably cynical about peer review- even those that serve as reviewers and editors- except maybe junior scientists that haven’t gotten burned yet.

I don't know how scientists handle peer review but aren't they fighting with peer review to get their papers published and apply for PhD and tenure and grants etc with these publications?

Re: Replace peer review with “peer replication” (2021)

#228

Post by tnecniv »

Maxion wrote: post #143
casualscience wrote: post #111

Earlier quoted context omitted.

Most of this is very legit, but this > Still other outsiders, who misunderstand the entire process, are upset that intermediate results are sometimes incorrect. This confuses them, and they’re angry that the process sometimes assigns “points” to people who they perceive as undeserving. So instead of simply accepting that sharing results widely to maximize the chance of verification is the whole point of the publicati…

And on the extreme end you have the multi-decade longitudinal studies in epidemiology / biomedicine that would be more-or-less impossible to replicate.

I remember reading that some epidemiologists saw the wealth of new data from CoVID as a silver lining because of how few events there are at that scale. Apparently it’s not uncommon to still use the Spanish Flu data which is spotty at best because it might be the only thing available at the scale you’re interested in

Re: Replace peer review with “peer replication” (2021)

#229

Post by indymike »

RugnirViking wrote: post #198
indymike wrote: post #188

Earlier quoted context omitted.

> 99% of all papers mean nothing. They add nothing to the collective knowledge of humanity. A lot of papers are done as a part of the process of getting a degree or keeping or getting job. The value is mostly the candidate showing they have the acumen to produce a paper of such quality that meets the publisher and peer review requirements. In some cases, it is to show a future employer some level of accomplishment or…

well yes. But these should go somewhere else than the papers that may actually contain significant results. The problem we have here is that there is an enormous quantity of such useless papers mixed in with the ones actually trying to do science. I understand that part of the reason for that is that people need to appear as though they are part of the "actually trying" crowd to get the desired job effects. But it is…

> I understand that part of the reason for that is that people need to appear as though they are part of the "actually trying" crowd to get the desired job effects

There's another less obvious problem: we don't always know what is groundbreaking and what is not until something is published and is accepted at large as being a really big deal.

Re: Replace peer review with “peer replication” (2021)

#230

Post by ramesh31 »

titzer wrote: post #86

In the PL field, conferences have started to allow authors to submit packaged artifacts (typically, source code, input data, training data, etc) that are evaluated separately, typically post-review. The artifacts are evaluated by a separate committee, usually graduate students. As usual, everything is volunteer. Even with explicit instructions, it is hard enough to even get the same code to run in a different environ…

> Even with explicit instructions, it is hard enough to even get the same code to run in a different environment and give the same results.

Is it really that hard for researchers to standardize around providing Dockerfiles? Environment replication is a solved problem.

Post reply on HN