https://web.archive.org/web/20230130143126/https://blog.ever...
Seems like article is not about software code.
11–20 of 351 posts
https://web.archive.org/web/20230130143126/https://blog.ever...
Seems like article is not about software code.
> the real test of a paper should be the ability to reproduce its findings in the real world. ... > What if all the experiments in the paper are too complicated to replicate? Then you can submit to [the Journal of Irreproducible Results]. Observational science is still a branch of science even if it's difficult or impossible to replicate. Consider the first photographs of a live giant squid in its natural habitat, pu…
Yes in a perfect world we would also replicate the data collection but we do not live in a perfect world
Same is true for Drug Trials, there is always a battle over getting the raw data from drug trails as the companies claim that data is trade secret, so independent verification of drug trails is very expensive but if the FDA required not just the release of redacted conclusions and supporting redacted data but 100% of all data gathered it would be alot better IMO
For example the FDA says it will take decades to release the raw data from the COVID Vaccine trials.. Why... and that is after being forced to do so via a law suit.
This of course depends a lot on the specific field, but it can easily be months of effort to replicate a paper. You save some time compared to the original as you don't have to repeat the dead ends and you might receive some samples and can skip parts of the preparation that way. But properly replicating a paper will still be a lot of effort, especially when there are any issues and it doesn't work on the first try. Then you have to troubleshoot your experiments and make sure that no mistakes were made. That can add a lot of time to the process.
This is also all work that doesn't benefit the scientists replicating the paper. It only costs them money and time.
If someone cares enough about the work to build on it, they will replicate it anyway. And in that case they have a good incentive to spend the effort. If that works this will indirectly support the original paper even if the following papers don't specifically replicate the original results. Though this part is much more problematic if the following experiments fail, then this will likely remain entirely unpublished. But the solution here unfortunately isn't as simple as just publishing negative results, it take far more work to create a solid negative result than just trying the experiments and abandoning them if they're not promising.
Replicator: Do you know how much data I'll need to collect? 11,000 particpants followed across multiple timepoints of MRI scanning. Show me the money.
How would you peer-replicate observation of a rare, or unique event, for example in astronomy?
Probably not gonna catch on.
Peer review does not serve to assure replication, but assure readability and comprehensibility of the paper. Given that some experiments cost billions to conduct, it is impossible to implement "Peer Replication" for all papers. What could be done is to add metadata about papers that were replicated.
If someone wants to spend some time replicating something that’s only been described in a paper or two, that is valuable work for the community and should be encouraged. If the person is a PhD student using that as an opportunity to hone their skills, it’s even better. It’s not glamorous, it’s not something entirely new, but it is useful and important. And this work needs to go to normal journals, otherwise there’s just be journals dedicated to replication and their impact factor will be terrible and nobody will care.
1. Rebrand peer review as a "readability review" which is what reviewers tend to focus on today.
2. A "replicability statement", a separately published document where reviewers push authors to go into detail about the methodology and strategy used to perform the experiments, including specifics that someone outside of their specialty may not know. Credit NalNezumi ITT