I've always been amazed by how widely the Stanford Prison Experiment results are accepted when a) the experiment has not been repeated and b) the experiment didn't even get completed. It was stopped when the researchers had made up their minds about the results.
We Should Not Accept Scientific Results That Have Not Been Repeated
101–110 of 282 posts
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#102This problem, like many in modern day science, can in large part be traced back to unstable funding. On the Maslow's-style hierarchy of research lab needs, the need for funding is a lot lower on the scale than the aspiration for scientific purity, just as a human's need for food is lower on the scale than their desire for self-actualization.
If competition for research dollars ceases to be so cutthroat, it will go a long way towards solving this and many other seemingly entrenched cultural problems.
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#103Define repetition. It's not as simple as that, for all sciences - once again an article on repeatability seems to have focused on medicinal drug research (it's usually that or psychology), and labelled the entire "Scientific community" as 'rampant' with " statistical, technical, and psychological biases". How about, Physics? The LHC has only been built once - it is the only accelerator we have that has seen the Higgs…
We don't need to 'define repetition', we need to foster a culture that a) accepts repetition and b) does not accept something for a fact just because it's in a journal. Right now, (a) is not even acceptable; nobody will publish a replication study. Ofc, LHC is impossible to replicate, but the vast majority of life science studies are. I should think this is mostly needed in life sciences. Other, more 'exact' sciences…
> we need to foster a culture that a) accepts repetition
Do we?
I don't think we do. I think we need to foster a culture of honesty and rigor. Of good science. Which is decidedly different from fostering a culture of "repetition" for its own sake.
Paying for the cost of mountains upon mountains of lab techs and materials that it would require to replicate every study published in a major journal just isn't a good use of ever-dwindling science dollars. Replicate where it's not far off the critical path. Replicate where the study is going to have a profound effect on the direction of research in several labs. But don't just replicate because "science!"
In fact, one could argue that the increased strain on funding sources introduced by the huge cost of reproducing a bunch of stuff would increase the cut-throat culture of science and thereby decrease the scientist's natural proclivity toward honesty.
> and b) does not accept something for a fact just because it's in a journal
Again, it's entirely unclear what you mean here.
It's impossible to re-verify every single paper you read (I've read three since breakfast). That would be like re-writing every single line of code of every dependency you pull into a project.
And I'm pretty sure literally no scientist takes a paper's own description of its results at face value without reading through methods and looking at (at least) a summary of the data.
Taking papers at face value is really only a problem in science reporting and at (very) sub-par institutions/venues.
I don't care about the latter, and neither should you.
WRT the former, science reporters often grossly misunderstand the paper anyways. All the good reproducible science in the world is of zero help if science reporters are going to bastardize the results beyond recognition anyways...
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#104We shouldn't "accept" or "reject" results at all. It's not a binary option. One poor experiment might give us some evidence something is true. A single well reviewed experiment gives us more confidence. Repeating the results similarly does. As does the reputation of the person conducting the experiment and the way in which it was conducted. It's not a binary thing where we decide something is accepted or rejected, we…
So many scientists I talk to don't have a basic understanding of philosophy of science. I don't necessarily blame them--I understand why "philosophy" as an academic field is seen as a soft, speculative, and pretentious field compared to the rigor of science, but as Daniel Dennett said, “There is no such thing as philosophy-free science; there is only science whose philosophical baggage is taken on board without examination."
These days, if you ask a scientist "So how do we prove something is true using science?" they'll be able to recite Popper's falsificationism as if it's a fundamental truth, not a particular way of looking at the world. But the huge gap between the particular theory that people get taught in undergrad--that science can't actually prove anything true, just disprove things to approach better hypotheses--and the real-world process of running an experiment, analyzing data, and publishing a paper is unaddressed. The idea that there's a particular bar that must be passed before we accept something as true is exactly what got us into this mess in the first place! There's a naive implicit assumption in scientific publishing that a p-value What's needed, in my opinion at least, is a more existential, practically-grounded view of science, in which we are more agnostic about the "truth" of our models with a closer eye to what we should actually do given the data. Instead of worrying about whether or not a particular model is "true" or "false," and thus whether we should "accept" or "reject" an experiment, focus on the predictions that can be made from the total data given, and the way we should actually live based on the datapoints collected. Instead, we have situations like the terrible state of debate on global warming, because any decent scientist knows they shouldn't say they're absolutely sure it's happening, or a replication crisis caused by experiments focused on propping up a larger model, instead of standing on their own.
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#105The big issue right now is funding of replicated research, who wants to fund a research to prove someone else was right? Most of these funds are granted based on potential outcome of the new discovery like: potential business, patents, licenses, etc... not being the first one would probably wipe most of these benefits, cutting down to a small probably getting funded... Now, straight to the point, who's going to pay f…
Perhaps those who are skeptical of the first research (and are losing money from it) should fund the replication research. Incentive for corrupting the data seems high, however.
This works for some "controversial" researches, which can be dismantled. What about successful (or apparently) ones which may lead to tons of $$$ in return? Is there any value to pay for a research which is going to prove someone else was right ending in making their wallets fatter?
Sorry for the informal language, but makes things a little bit more salty.
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#106I think with the increased visibility of scientific research to the general public, it's less that science needs to stop accepting unrepeated results, but instead the paper process needs to be updated to reflect the new level of availability, and journal databases need better relationship views between papers and repeated tests. As an outsider looking in on the Scientific process, I am not really sure how applicable…
I think also it would be helpful if they listed all the possible flaws first, including things about if it has been replicated etc.
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#107Maybe conferences should have a "reproducibility" track for that purpose? Also, I don't know about other fields, but I'm pretty sure that in CS, if you just took a paper and tried to reproduce the results, you'll get rejected on the ground that you offer no original contribution; no original contribution => no publication => no funding.
For CS, reproducing should be easy given the code and input data right? Other sciences' input isn't so easily shared
I mean, if a paper gives an algorithm, proves its correctness, and you are convinced by the proofs, then you're done. I don't see how implementing the algorithm gives you more insight. I'm doing my master degree in computational geometry and most people in my lab don't even implement their algorithms. They just know they are correct from their proofs.
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#108Lottery < Statistics < Science
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#109Define repetition. It's not as simple as that, for all sciences - once again an article on repeatability seems to have focused on medicinal drug research (it's usually that or psychology), and labelled the entire "Scientific community" as 'rampant' with " statistical, technical, and psychological biases". How about, Physics? The LHC has only been built once - it is the only accelerator we have that has seen the Higgs…
With all respect I do not agree with your example for the lhc .
For two reasons primarily:
The first has already been mentioned about various groups and experiments / detectors.
The second, I think is more important. In order to claim discoveries at any of those experiments certain statistical constraints must be met. The 5 sigma rule is by its very nature garnered by repetition and is a necessary because of the nature of quantum mechanics and particle physics. Here is a write up on cern and 5 sigma
http://physicsbuzz.physicscentral.com/2012/07/does-5-sigma-d...
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#110Define repetition. It's not as simple as that, for all sciences - once again an article on repeatability seems to have focused on medicinal drug research (it's usually that or psychology), and labelled the entire "Scientific community" as 'rampant' with " statistical, technical, and psychological biases". How about, Physics? The LHC has only been built once - it is the only accelerator we have that has seen the Higgs…
We don't need to 'define repetition', we need to foster a culture that a) accepts repetition and b) does not accept something for a fact just because it's in a journal. Right now, (a) is not even acceptable; nobody will publish a replication study. Ofc, LHC is impossible to replicate, but the vast majority of life science studies are. I should think this is mostly needed in life sciences. Other, more 'exact' sciences…
I think there's also a certain lack of transparency in the scientific process that needs to be fixed. But that's just like my opinion.