Having wasted time trying to replicate someone else's results who 'lost' their code, I agree! Maybe repeating the experiment should be part of the peer review.
So frustrating. Lost = "I didn't think anyone would hold me accountable for it."
We Should Not Accept Scientific Results That Have Not Been Repeated
91–100 of 282 posts
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#92Earlier quoted context omitted.
I don't think reproducibility should necessarily mean literally repeating the same study over again. Whether or not a paper is fully correct is less important that the further work it stimulates or informs-- either via more papers (impact factor) or via APPLICATIONS of the research. In the example you're giving, to get accepted, the author doesn't need to do much more than extend the research in some small way, compa…
"I don't think reproducibility should necessarily mean literally repeating the same study over again." Yes, it should. We have too many demonstrated instances of studies built on other studies that turned out to be flawed, but the second study, rather than showing the first one was flawed, was rationalized and massaged until it conformed to the first study because the first study already had the imprimatur of peer-re…
But there should also be replication of the concepts of the study under a different paradigm, otherwise you might just be finding out effects of the particular procedure. An original study is rarely "wrong" (fraud is a serious problem, but not a typical one), it's that it's merely one data point, from one procedure, and a lot of the time the effect that can be replicated is "real," it's just a side effect of the process. Basically, there's way more replication that should be going on in science at many levels.
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#93Sometimes in cs if your research is embedded in a huge ecosystem, it can become quite expensive to reproduce results. I mean proper reproduction, not just rerunning the Benchmarks. If you are dealing with complicated stuff, the reproducer might also just not be able to do the same thing technically.
Maybe, maybe not. Do you have something specific in mind here? I hope researchers and scientists don't considers others not capable enough, and therefore withhold info on how to reproduce. Even if the experiment is crazy expensive and complex right now it might be considered much more tractable in 10 years, or someone builds upon your work and invents a simpler method to show the same thing.
I am thinking of huge endeavors like building an asic or huge complex systems like virtual machines. Not always a comparable system for Repetition is available and must be built from scratch. Affording such rebuilds require huge sums.
Of course nobody does consider others not capable enough. Its just that there are not so many people experienced enough to build certain systems in a decent amount of time.
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#94Sometimes in cs if your research is embedded in a huge ecosystem, it can become quite expensive to reproduce results. I mean proper reproduction, not just rerunning the Benchmarks. If you are dealing with complicated stuff, the reproducer might also just not be able to do the same thing technically.
Maybe, maybe not. Do you have something specific in mind here? I hope researchers and scientists don't considers others not capable enough, and therefore withhold info on how to reproduce. Even if the experiment is crazy expensive and complex right now it might be considered much more tractable in 10 years, or someone builds upon your work and invents a simpler method to show the same thing.
[deleted]
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#95This is wrong-headed in the extreme.
What we should demand is scientific results that have FAILED.
When we see a p=0.05, but we don't know that this SAME EXACT EXPERIMENT has been run 20 times before, we're really screwing ourselves over.
Relevant: https://xkcd.com/882/
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#96Sometimes in cs if your research is embedded in a huge ecosystem, it can become quite expensive to reproduce results. I mean proper reproduction, not just rerunning the Benchmarks. If you are dealing with complicated stuff, the reproducer might also just not be able to do the same thing technically.
Just being able to rerun the benchmarks (or other data analysis for non-CS papers) would be an improvement on the current state of affairs, where people often don't publish code nor data.
Agreed. Some cs conferences verify artifacts these days, which is a good first step.
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#97We shouldn't "accept" or "reject" results at all. It's not a binary option. One poor experiment might give us some evidence something is true. A single well reviewed experiment gives us more confidence. Repeating the results similarly does. As does the reputation of the person conducting the experiment and the way in which it was conducted. It's not a binary thing where we decide something is accepted or rejected, we…
^ One of the only reasonable responses here.
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#98Just to play devils advocate, won't there be a self correcting mechanism?
If results are genuinely useful, then people will want to build upon that work, and will have to repeat the science. On the other hand if it can't be repeated, then it will not get further work done and fade into obscurity. Curious what other peoples opinion on this are?
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#99Earlier quoted context omitted.
I'm pointing out that it's impossible to work this way. People don't like it. The accounting systems aren't set up for this. The entire post-war research system isn't structured this way.
The entire post-war research system is set up to create endless streams of non-replicable studies based on incorrectly-applied statistics all the while convincing itself that it's doing something useful while wasting billions upon billions of dollars. I'm not to impressed by "well, that's not how it works right now". The whole problem is "how it works right now". That's what we're discussing, the need for it to not w…
You want to change the system. I understand that.
We have many systems to go on over the last few hundred years of science. We have the pre-war system, primarily funded by private philanthropy. We have the communist system.
None of them seem to create the stream of highly replicable studies you want.
That may indicate something deep about how people work and how science is really done, and suggest that your admirable goals are not tenable.
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#100[deleted]