Replace peer review with “peer replication” (2021)
151–160 of 351 posts
Re: Replace peer review with “peer replication” (2021)
#152Peer review does not serve to assure replication, but assure readability and comprehensibility of the paper. Given that some experiments cost billions to conduct, it is impossible to implement "Peer Replication" for all papers. What could be done is to add metadata about papers that were replicated.
>Experiments that cost billions to conduct If you can't replicate them it's like they didn't happen anyways
Ah yes, if I can’t run the LHC at home, none of the work there happened
Re: Replace peer review with “peer replication” (2021)
#153I wish we can replicate the LHC
No talking about the Higgs before that happens, apparently.
Re: Replace peer review with “peer replication” (2021)
#154I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…
>I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. I think it would be fine to half the productivity of these fields, if it means that you can reasonably expect papers to be accurate.
The issue that I see is: even if halving productivity is acceptable to the field as a whole; how do you incentivize a given scientist to put in the effort?
This seems particularly problematic because it is already notoriously hard to get tenure and academia is already notoriously unrewarding to researchers who don't have tenure.
Re: Replace peer review with “peer replication” (2021)
#155I spent a lot of my graduate years in CS implementing the details of papers only to learn that, time and time again, the paper failed to mention all the short comings and fail cases of the techniques. There are great exceptions to this. Due to the pressure of "publish or die" there is very little honesty in research. Fortunately there are some who are transparent with their work. But for the most part, science is dro…
I had a very similar experience in my masters. Really made me think, what exactly are the peers “reviewing” if they don’t even know whether the technique works in the first place.
I have reviewed many papers and there is never the time to recreate the work and test. That is why I love the "papers w code" site. I think every published CS paper should require a git repo with all their code and experimental data.
Re: Replace peer review with “peer replication” (2021)
#156I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…
> I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. They also tend to over-estimate the effect of peer review (often equating peer review with validity). > If someone cares enough about the work to build on it, they will replicate it anyway. And in that case they have a good incentive to spend the effort…
“They also tend to over-estimate the effect of peer review (often equating peer review with validity).“
In my experience, scientists ate comfortably cynical about peer review- even those that serve as reviewers and editors- except maybe junior scientists that haven’t gotten burned yet.
Re: Replace peer review with “peer replication” (2021)
#157I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…
It's simple but not easy: You create another path to tenure which is based on replication, or on equal terms as a part of a tenure package. (For example, x fewer papers but x number of replications, and you are expected to have x replications in your specialty.) You also create a grant funding section for replication which is then passed on to these independent systems. (You would have to have some sort of randomizat…
I think that there's also a lot of psychological/cultural/political issues that work also need to be worked out:
If someone wins the Nobel Prize, do the people who replicated their work also win it? When the history books are written do the replicators get equal billing to the people who made the discovery?
When selecting candidates for prestigious positions, are they really going to consider a replicator equal to an original researcher?
Re: Replace peer review with “peer replication” (2021)
#158Earlier quoted context omitted.
Fraud seems rampant because you hear about cases of fraud, but not about the tens of thousands of research labs plugging away day after day.
I agree that most labs are probably not out to defraud people. But without replication I don't think it's reasonable to have much confidence in what is published.
replication happens over time.
For example, when I did my PhD I wanted to grow TaS2 monolayers on a graphene layer on an Iridium crystal. So I took published growth recipees of related materials, adapted them to our setup and then finetuned the recipee for TaS2. This way I basically "peer replicated" the growth of the original paper. I then took those samples to a measurement device and modified the sample in-situ by evaporating Li atoms on top (which was the actual paper but I needed a sample to modify first). I published the paper with the growth recipee and the modification procedure and other colleagues then took those instructions to grow their own samples for their own studies (I think it was MoS2 on Graphene on Cobalt that they grew).
This way papers are peer replicated in an emerging manner because the knowledge is passed from one group to another and they use parts of that knowledge to then apply it to their own research. You have to see this from a more holistic picture. Individual papers don't mean too much, it's their overlap that generates scientific consesus.
In contrast, requiring some random reviewer to instead replicate my full paper would be an impossible task. He/she would not have the required equipment (because there's only 2 lab setups in the whole world with the necessary equipment), he/she would probably not have the required knowledge (because mine and his research only partially overlap - e.g. we're researching the same materials but I use angle-resolved photoemission experiments and he's doing electronic transport) and he/she would need to spend weeks first adapting the growth recipee to the point where his sample quality is the same as mine.
Re: Replace peer review with “peer replication” (2021)
#159As much as I agree with the sentiment, we have to admit it isn't always practical. There's only one LIGO, LHC or JWST, for example. Similarly, not every lab has the resources or know-how to host multi-TB datasets for the general public to pick through, even if they wanted to. I sure didn't when I was a grad student.
That said, it infuriates me to no end when I read a Phys. Rev. paper that consists of a computational study of a particular physical system, and the only replicability information provided is the governing equation and a vague description of the numerical technique. No discretized example, no algorithm, and sure as hell no code repository. I'm sure other fields have this too. The only motivation I see for this behavior is the desire for a monopoly on the research topic on the part of authors, or embarrassment by poor code quality (real or perceived).
Re: Replace peer review with “peer replication” (2021)
#160Earlier quoted context omitted.
Who will be replicating it? Why would I want to set aside my own research to replicate some claim someone made? How would this help my career?
Academia's values are not objective. Why is it that replicating or refuting a study is not seen on par as being a co-author of said study? There is nothing set in stone preventing this, only the current academic culture.
Because I want to do original research, and be known for doing original research. Only if I fail at that, I might settle for being a guy who reproduces others’ work (which basically means the transition from a researcher to an engineer).