this would not apply to math or something subjective such as literature. only experimental results need to be replicated.
Replace peer review with “peer replication” (2021)
131–140 of 351 posts
Re: Replace peer review with “peer replication” (2021)
#132Earlier quoted context omitted.
What about a system where peer replication is required if the number of citations exceeds some threshold?
Who will be replicating it? Why would I want to set aside my own research to replicate some claim someone made? How would this help my career?
Academia's values are not objective. Why is it that replicating or refuting a study is not seen on par as being a co-author of said study? There is nothing set in stone preventing this, only the current academic culture.
Re: Replace peer review with “peer replication” (2021)
#133I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…
>I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. I think it would be fine to half the productivity of these fields, if it means that you can reasonably expect papers to be accurate.
I believe that, contrary to popular belief, the implementation of this system would lead to a substantial increase in productivity in the long run. Here's why:
Currently, a significant proportion of research results in various fields cannot be reproduced. This essentially means that a lot of work turns out to be flawed, leading to wasted efforts (you can refer to the 'reproducibility crisis' for more context). Moreover, future research often builds upon this erroneous information, wasting even more resources. As a result, academic journals get cluttered with substandard work, making them increasingly difficult to monitor and comprehend. Additionally, the overall quality of written communication deteriorates as emphasis shifts from the accurate transfer and reproduction of knowledge to the inflated portrayal of novelty.
Now consider a scenario where 50% of all research is dedicated to reproduction. Although this may seem to decelerate progress in the short term, it ensures a more consistent and reliable advancement in the long term. The quality of writing would likely improve to facilitate replication. Furthermore, research methodology would be disseminated more quickly, enhancing overall research effectiveness.
Re: Replace peer review with “peer replication” (2021)
#134Earlier quoted context omitted.
> If someone cares enough about the work to build on it, they will replicate it anyway. Does it really deserve to be called work if it doesn't include the a full, working set of instructions that if followed to a T allow it to be replicated? To me that's more like pollution, making it someone else's problem. I certainly don't see how "we did this, just trust us" can even be considered science, and that's not because…
I work with code, which is about as reproducible as it is possible to get - the artifacts I produce are literally just instructions on how to reproduce the work I've done again, and again, and again. And still people come to me with some bug that they've experienced on their machine, that I cannot reproduce on my machine, despite the two environments being as identical as I can possibly make them. I agree that reprod…
> And still people come to me with some bug that they've experienced on their machine, that I cannot reproduce on my machine
But this is the other way around. Have you ever written a program that doesn't run anywhere except a single machine of yours? Would you release it and advertise it and encourage other people to use it as dependency in their software?
If it only runs on one machine of yours, you don't even know if your code is doing something, or something else in the machine/OS. Or in terms of science, whether the research says something about the world, or just about the research setup.
Re: Replace peer review with “peer replication” (2021)
#135I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…
Longer, even! Some experiments that study biological development or trained animals can take a year or more of fairly intense effort to start generating data.
A year? some data sets take decades to build up before significant papers can be published on their data. Replication of the dataset is just not feasible.
This whole thread just shows how little the average HNer knows about the academic sciences.
Re: Replace peer review with “peer replication” (2021)
#136Earlier quoted context omitted.
Fraud seems rampant because you hear about cases of fraud, but not about the tens of thousands of research labs plugging away day after day.
I agree that most labs are probably not out to defraud people. But without replication I don't think it's reasonable to have much confidence in what is published.
Maybe replication should (and probably does) happen when the published thing is relevant to some entity and also interesting.
I never seen papers as "truth", but more as "possibilities". After many other "proofs" (products, papers, demos, etc.) you can assign some concepts/ideas the label "truth" but one/two papers from the same group is definitely not enough.
Re: Replace peer review with “peer replication” (2021)
#137The website dies if I try to figure out who the author (“sam”) is, but it sounds like they are used to some awful backwater of academia.
They have this idea that a single editor screens papers to decide if they are uninteresting or fundamentally flawed, then they want a bunch of professors to do grunt work litigating the correctness of the experiments.
In modern (post industrial revolution) branches of science, the work of determining what is worthy of publication is distributed amongst a program committee, which is comprised of reviewers. The editor / conference organizers pick the program committee. There are typically dozens of program committee members, and authors and reviewers both disclose conflicts. Also, papers are anonymized, so the people that see the author list are not involved in accept/reject decisions.
This mostly eliminates the problem where work is suppressed for political reasons, etc.
It is increasingly common for paper PDFs to be annotated with badges showing the level of reproducibility of the work, and papers can win awards for being highly reproducible. The people that check reproducibility simply execute directions from a separate reproducibility submission that is produced after the paper is accepted.
I argue the above approach is about 100 years ahead of what the blog post is suggesting.
Ideally, we would tie federal funding to double blind review and venues with program committees, and papers selected by editors would not count toward tenure at universities that receive public funding.
Re: Replace peer review with “peer replication” (2021)
#138I don't see how this could ever work, and non-scientists seem to often dramatically underestimate the amount of work it would be to replicate every published paper. This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of…
Longer, even! Some experiments that study biological development or trained animals can take a year or more of fairly intense effort to start generating data.
I know people that had to take a 6+ month trip to Antarctica for part of their work and others that had to share time on a piece of experimental equipment with a whole department — they got a few weeks per year to run their experiment and had to milk that for all it’s worth. Even if they had funding, that machine required large amounts of space and staff to keep it running and they aren’t off the shelf products — only a few exist at large research centers.
Re: Replace peer review with “peer replication” (2021)
#139Earlier quoted context omitted.
I agree that most labs are probably not out to defraud people. But without replication I don't think it's reasonable to have much confidence in what is published.
That's not what publication is about. Publication is a conversation with other researchers; it is part of the process of reaching the truth, not its endpoint.
People in general (at least on da Internetz) seem to focus way to much on single studies, and way too little on meta-studies.
AFAICT meta-studies is the level where we as a society really can try to say something intelligent about how stuff works. If an important question is not included in a meta-study, we (i.e. universities and research labs) probably need to do more research on that topic before we really can say that much about it.
Re: Replace peer review with “peer replication” (2021)
#140[flagged]