Agreed, which means 50% of social science at least is disqualified and should not be making into future publications or become part of curriculum.
We Should Not Accept Scientific Results That Have Not Been Repeated
191–200 of 282 posts
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#192Define repetition. It's not as simple as that, for all sciences - once again an article on repeatability seems to have focused on medicinal drug research (it's usually that or psychology), and labelled the entire "Scientific community" as 'rampant' with " statistical, technical, and psychological biases". How about, Physics? The LHC has only been built once - it is the only accelerator we have that has seen the Higgs…
You ask that like it is something new. Reproducibility[0] is something at the core of science. The problem these days is that spending time doing replication is not glamourous and will not help you get funds. [0]: https://en.wikipedia.org/wiki/Reproducibility
The problem is worst in fields that don't have reproducibility 'built in' to the field.
I do genetics and development, and the main sanity check we have is the distribution of mutant lines. If you say that mutant X does Y, other people are likely going to see that (or not) when they get their hands on it and start poking around. This strength of working with mutants is at the core of the success of molecular biology. Even if don't set out to confirm someone else's results, you're quite likely to come across some inconsistencies in the course of investigation.
If a field lacks that sort of mechanism, they need to take special care to address reproducibility.
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#193I agree with the main point of this article but in terms of its analysis and prescriptions I think it gets two things backwards. (1) Most scientists seek fame as a means to the end of getting tenure and funding, not the other way around; if you gave them tenure (and the ability to move their tenure to somewhere else if they wanted to move) and perpetual funding and told them they could choose to be anonymous, I think many would choose that option. (2) Replication is not done/published enough because the incentive to do so (measured in: increase in probability of getting tenure per hour spent) is not high enough, not because people are overly willing to accept unreplicated work.
In order for a lot more replication to get published, what would be needed would be for people who spent their careers replicating others' results (at the expense of not producing any important novel results of their own) to get tenure at top institutions (outcompeting others who had important novel results but not enough published replications).
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#194Earlier quoted context omitted.
We don't need to 'define repetition', we need to foster a culture that a) accepts repetition and b) does not accept something for a fact just because it's in a journal. Right now, (a) is not even acceptable; nobody will publish a replication study. Ofc, LHC is impossible to replicate, but the vast majority of life science studies are. I should think this is mostly needed in life sciences. Other, more 'exact' sciences…
Does doing 50 studies and only publishing the 3 that produce favorable results count as repetition? Because, BTW, that's exactly what Big Pharma does, and also intrinsically what the bias towards publishing only significant results does.
Pharma needs to send information about studies beforehand and regulators pay attention to this for new drugs which counters the effect your talking about. Further, they are happy to report inconclusive results the FDA mostly cares about harm to people not effectiveness.
Now, after drugs are on the market then there is another wave of less reputable research. But, the FDA has already approved the drug so they don't care as much.
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#195> Nowadays there's a certain danger of the same thing happening (not repeating experiments), even in the famous field of physics. I was shocked to hear of an experiment done at the big accelerator at the National Accelerator Laboratory, where a person used deuterium. In order to compare his heavy hydrogen results to what might happen with light hydrogen, he had to use data from someone else's experiment on light, hyd…
Current top comment https://news.ycombinator.com/item?id=12186295 , specifically rebutted 30 years before the comment was written.
This isn't a rebuttal of the linked comment.
The linked comment doesn't state that it would be a waste of time to replicate on a hypothetical LHC clone.
Rather, the linked comment states that we can accept the Higgs result with reasonable confidence even though it's currently infeasible to replicate that experiment.
Feynman's issue was also qualitiatively different -- the scientist was comparing results from two different instruments. The people in charge of one of the instruments wouldn't allow the scientist to run both experiments on a single instrument. In fact, from context, it's not even clear to me Feynmann would have insisted on re-running the original experiment if the scientist were not using a different accelerator for the second one. Anyways, in the Higgs case, there's no potential for a "comparing readings from instrument A to readings from instrument B" type bug.
More to the point, and FWIW, I somehow doubt Feynman would insist on building a second LHC for the sole purpose of replicating the Higgs measurement. But I guess we have to leave that to pure speculation.
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#196I see a lot of people commenting here that there's no incentive to repeat previous research because it's not useful for getting grants, etc. This kind of true but I think it misses something important. At least in life sciences (can't comment on other fields), it's not that scientists don't repeat each other's results. After all, if you're going to invest a significant fraction of your tiny lab budget on a research p…
>>If the replication fails, it's even more time-consuming and expensive to convincingly demonstrate the negative result.
Said scientist will gain immense fame and recognition if they show established ideas are wrong. (Source: academia Physicist)
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#197I see a lot of people commenting here that there's no incentive to repeat previous research because it's not useful for getting grants, etc. This kind of true but I think it misses something important. At least in life sciences (can't comment on other fields), it's not that scientists don't repeat each other's results. After all, if you're going to invest a significant fraction of your tiny lab budget on a research p…
This is why I'm always skeptical of people zealously claiming "all GMOs are perfectly safe! It's been proven !" Yeah, proven by expensive studies that were funded by the company making the GMO. Who is going to pay for another study to try to disprove it? There's a reason we sprayed DET all over our vegetables for years before it was banned: there was no scientific studies proving that it was harmful, even though it c…
Do you mean DDT?
Anyway, beyond the 'philosophy of science' issue of whether you can prove something, there is good affirmative evidence that existing GMOs are safe for numerous reasons.
First, there's no mechanistic reason to think they would be dangerous. T-DNAs are not somehow magically toxic to humans; everything you eat is riddled with millennia of t-dnas, old viruses, transposon blooms, etc. etc.
The technologies themselves should be safe as well. BT is well understood and embraced by the organic community as an applied natural pesticide, so you would need to find evidence that the localization somehow makes it toxic. Glyphosate resistance is also unlikely to have any effect a priori because it affects a metabolic pathway completely absent in animals.
Argue all you like about how nothing can be 'perfectly safe', sure, but there's no reason to think that GMOs are dangerous, and people have looked quite hard. ______
Finally, just look at the Seralini pile-of-bullshit for evidence that there's plenty of incentive to publish anything critical of GMOs. No one is sitting on career-making evidence.
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#198Earlier quoted context omitted.
Are you for real? You would sacrifice people upon the alter of reproducibility? Down that path lies atrocities. The system was put into place to prevent repeats of horrors like the "Tuskegee Study of Untreated Syphilis in the Negro Male".
I'd rather not sacrifice people on the altar of a single study, no matter how significant the results. Down that path lies atrocities, too, albeit of a quieter sort.
As I said earlier, there are alternatives which are both moral and can verify effectiveness without having to repeat the original experiment.
You chose to not verify, and insist upon repeating, thus likely consigning people to unneeded pain and even death.
I'll give a real-world example to be more clear cut about modern ethics and science. Ever hear of TGN1412? https://en.wikipedia.org/wiki/TGN1412
It went into early human trials, and very quickly caused a reaction. "After very first infusion of a dose 500 times smaller than that found safe in animal studies, all six human volunteers faced life-threatening conditions involving multiorgan failure for which they were moved to intensive care unit." (http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2964774/ )
Here's a publication of the effects: http://www.nejm.org/doi/full/10.1056/NEJMoa063842 .
Is it moral to reproduce that experiment? I say it is not moral, and must not be repeated even though it is possible to do so.
Can a publication about the effects still be good science even though medical ethics prevent us from repeating the experiment? Absolutely.
What say you?
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#199I see a lot of people commenting here that there's no incentive to repeat previous research because it's not useful for getting grants, etc. This kind of true but I think it misses something important. At least in life sciences (can't comment on other fields), it's not that scientists don't repeat each other's results. After all, if you're going to invest a significant fraction of your tiny lab budget on a research p…
>>If the replication fails, it's even more time-consuming and expensive to convincingly demonstrate the negative result. Said scientist will gain immense fame and recognition if they show established ideas are wrong. (Source: academia Physicist)
I think your's and the parent comment are both right, just addressing different extremes.
Sometimes you find evidence that doesn't quite match up with previous work, but if an explanatory framework isn't forthcoming, it can be difficult to say who's results are right, and what they mean. On the other hand, sometimes you come across something blatantly wrong, so you do your follow up experiments to confirm and you're all set. It's only the latter case that will quickly make it into a manuscript.
I will say, I've seen some big corrections (like this-allele-was-actually-a-completely-different-mutant bad) that just get buried in the results section of a subsequent publication, and no retraction was ever submitted for the original paper. That was definitely a failure in the field, and likely the result of the status of the authors involved.
Re: We Should Not Accept Scientific Results That Have Not Been Repeated
#200I'm a physician and I've been suggesting this to my colleagues for a few years, only to be met with alienated stares and labeled cynical. The doctors and doctors-in-training I work with have altruistic motives, but place too much stock in major medical studies. They also frequently apply single-study findings to patient care, even to patients that would've been excluded from that study (saw this a lot with the recent…
Personal experience (I've met many pre- and current doctors during my studies) suggests that few doctors are scientists . They know a lot of things, but they're not what I'd call scientists in the same way a physicist or biologist would be (and I've often wondered why they'd get a "Doctorate" at all). So this might explain the abovementioned behaviour... Yet I'm surprised by your colleagues behaviour nonetheless. I w…
Aside from MD/PhD's, medical doctors are not trained in research or evaluating evidence, so are not scientists by any reasonable definition.