Having wasted time trying to replicate someone else's results who 'lost' their code, I agree! Maybe repeating the experiment should be part of the peer review.
We Should Not Accept Scientific Results That Have Not Been Repeated
51–60 of 282 posts
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#52My first reaction to this headline was "duh." Of course we should hold off on accepting scientific claims (i.e., predictions about the natural world) that to date have been verified only by the same person making those claims! My next reaction was, "wow, it's a sad state of affairs when a postdoctoral research fellow at Harvard Medical School feels he has to spell this out in a blog post." It implies that even at the…
Of course you as a reader of said claims confirm at the very least that they've been independently reproduced, right? (If so, this shouldn't be news.)
Unfortunately, there is no easy way to do it. Confirmation studies are not easily accepted by impactful journals/conferences, thus nearly nobody bothers to do them. Even if there is one, it can be surprisingly hard to find it.
As a point of anecdata: my wife's master thesis was a confirmation study of using LLDA for face recognition. I remember seeing it included in some book by the university press. I gave up Googling for it after 5 minutes.
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#53I agree in principle. There are a few concerns:
1. How should we receive costly research that took special equipment and lots of time to develop and cultivate? I.e., CERN?
2. A lot of research is published, ignored, and then rediscovered. In this case, we may want to accept the research until it cannot be repeated (i.e., in another journal publication).
3. Reviewers of academic publications probably are not qualified or have the time to recreate all scientific research.
4. Isn't the academic system at its core kinda... broken?
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#54Totally agree. I'd go even further and make free licenses on scientific source and datasets mandatory. Research that is funded by public money should lead to public code and data.
So, $1 of public funding triggers public code release? Or is there a threshold?
No one has been able to tell me why the need for reproducibility requires software freedom.
Consider the program 'nauty'. It is available in source code form for anyone to review, but it cannot be used for military purposes. That's not free, certainly. But isn't that enough to call it good science?
Similarly, consider the clause "only for use in verifying the result of paper X". That's also not free. But it serves the goal of letting others be able to verify X.
Also, you haven't gone far enough. It's not only the license that matters, but access. You have to mandate that either anyone can get access to the code for no/low cost for some years (since I can sell my GPL'ed software for $30,000 or take down the download link once published), or link publication with a required submission to some repository with the mission of keeping all that source and data around, available to anyone, at no cost.
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#55I agree in principle. There are a few concerns: 1. How should we receive costly research that took special equipment and lots of time to develop and cultivate? I.e., CERN? 2. A lot of research is published, ignored, and then rediscovered. In this case, we may want to accept the research until it cannot be repeated (i.e., in another journal publication). 3. Reviewers of academic publications probably are not qualified…
"Isn't the academic system at its core kinda... broken?"
can your elaborate on what you mean?
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#56Earlier quoted context omitted.
but in medicine, what is the practical alternative? how do you incorporate these findings? ignore them? if so, it's probably bad for your patients. the only thing worse than a single-study finding is a zero-study finding.
I was simply suggesting what we all learn in medical school and residency: to appropriately evaluate clinical studies. Just don't think most doctors do. Let me give you an example of how I approach things. The guidelines for acute pancreatitis recommend using a fluid called LR instead of NS for volume resuscitation. This is based on an single study that included 10 patients and simply noted slightly better lab number…
"Probably" is one of favorite words in medicine, btw :).
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#57Earlier quoted context omitted.
Broken assumes it was working well at some point. Nothing new here.
It worked pretty well back when scientific inquiry was funded by selling inventions or other work that the scientist did "as a day job." That was a long time ago, though.
This has worked and still works for technology, but is not how basic science research was done. There's never been "enough" money for sciences, nor they ever been allocated in a manner agreeable to all parties. For a long time, research was either a gentleman's hobby or a side project under auspices of unsuspecting donor. Lagrange has developed variational calculus while doing his artillery school tenure. Einstein's annus mirabilis happened at his stint in Swiss patent office.
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#58I agree in principle. There are a few concerns: 1. How should we receive costly research that took special equipment and lots of time to develop and cultivate? I.e., CERN? 2. A lot of research is published, ignored, and then rediscovered. In this case, we may want to accept the research until it cannot be repeated (i.e., in another journal publication). 3. Reviewers of academic publications probably are not qualified…
[deleted]
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#59Define repetition.
It's not as simple as that, for all sciences - once again an article on repeatability seems to have focused on medicinal drug research (it's usually that or psychology), and labelled the entire "Scientific community" as 'rampant' with " statistical, technical, and psychological biases".
How about, Physics?
The LHC has only been built once - it is the only accelerator we have that has seen the Higgs boson. The confirmation between ATLAS and CMS could be interpreted as merely internal cross-referencing - it is still using the same acceleration source. But everyone believes the results, and believes that they represent the Higgs. This isn't observed once in the experiment, it is observed many, many times, and very large amounts of scientists time are spent imagining, looking for, and measuring, any possible effect that could cause a distortion or bias to the data. When it costs billions to construct your experiment, sometimes reproducing the exact same thing can be hard.
The same lengths are gone to in order to find alternate explanations or interpretations of the result data. If they don't, they know that some very hard questions are going to be asked - and there will be hard questions asked anyway, especially for extraordinary claims - look at e.g. DAMA/LIBRA which for years has observed what looks like indirect evidence for dark matter, but very few people actually believe it - the results remain unexplained whilst other experiments probe the same regions in different ways.
Repetition is good, of course, but isn't a replacement for good science in the first place.
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#60Earlier quoted context omitted.
For CS, reproducing should be easy given the code and input data right? Other sciences' input isn't so easily shared
Ideally you should re-implement the algorithm based on the description in the paper to verify that the description of the algorithm is correct. You should also test with your own data to make sure that the algorithm works on all reasonable data and not only on some provided cherry picked data. If you can't get the expected results with your own implementation and your own data then the results aren't reproduced.
Yes. So being able to rerun with the same code and same inputs to get the same outputs is a lower bar. Many papers don't meet even that bar.
(Mostly because they don't publish code nor data; and academic code is often a horrible mess, and the code was mucked around with between different stages of running.)