Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Should Not Accept Scientific Results That Have Not Been Repeated

nautil.us

Post reply on HN
Print view View original

151–160 of 282 posts

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#151

Post by misnome »

welterde wrote: post #83
misnome wrote: post #59

Define repetition. It's not as simple as that, for all sciences - once again an article on repeatability seems to have focused on medicinal drug research (it's usually that or psychology), and labelled the entire "Scientific community" as 'rampant' with " statistical, technical, and psychological biases". How about, Physics? The LHC has only been built once - it is the only accelerator we have that has seen the Higgs…

ATLAS, CMS, ALICE, LHCb, etc. are all different experiments that just happen to share the same accelerator. They are done using different detectors, designed, built, operated and the data is analyzed by different groups of people. If you follow your logic, that the different experiments using the same accelerator negates the whole thing, to the extreme then doing two experiments on the same planet/solar system/univer…

I think you've got my argument completely backwards - I'm pointing out that if repetition as implied by the article is everything, then results from e.g. the LHC are discounted.

Similarly, I am not saying that you don't need to repeat - just that it isn't the be all and end all of what defines 'science'. Supporting this interpretation is that I mentioned DAMA. Nobody is accusing them of not taking care, but nobody really believes the result either.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#152

Post by peter303 »

misnome wrote: post #59

Define repetition. It's not as simple as that, for all sciences - once again an article on repeatability seems to have focused on medicinal drug research (it's usually that or psychology), and labelled the entire "Scientific community" as 'rampant' with " statistical, technical, and psychological biases". How about, Physics? The LHC has only been built once - it is the only accelerator we have that has seen the Higgs…

There were several "bogey Higgs" at different energies during earlier Tevatron and LHC runs. Maybe these were 3-5 events which hinted at a signal. Then they disappeared. But the real Higgs was not confirmed until there were 40 events of two different decay paths at two different detectors, i.e. 5-sigma significance. Different runs and decay paths are like multiple replications.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#153

Post by tcpekin »

selectron wrote: post #142
throwaway729 wrote: post #103

Earlier quoted context omitted.

> we need to foster a culture that a) accepts repetition Do we? I don't think we do. I think we need to foster a culture of honesty and rigor. Of good science. Which is decidedly different from fostering a culture of "repetition" for its own sake. Paying for the cost of mountains upon mountains of lab techs and materials that it would require to replicate every study published in a major journal just isn't a good use…

> Taking papers at face value is really only a problem in science reporting and at (very) sub-par institutions/venues. > WRT the former, science reporters often grossly misunderstand the paper anyways. All the good reproducible science in the world is of zero help if science reporters are going to bastardize the results beyond recognition anyways... Science is funded by the public, and done for the public. Good scien…

There's also the option that papers are written the way they are so that they still remain papers and not books. A five page paper on someone's findings is much easier to read than a 20-30 page paper where field specific knowledge is redefined and explained instead of referenced.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#154

Post by hx87 »

dalke wrote: post #63
ebbv wrote: post #44

Earlier quoted context omitted.

> One part of science is observation. Including observations which cannot be, or at least have not been, repeated. For example, consider a rare event in astronomy which has only been detected once. Is that science? I say it is. But it's surely not repeatable. First off, you can accept the observation at face value as an observation, but conclusions drawn from the claims which have no other support or means of verific…

> conclusions drawn from the claims which have no other support or means of verification should not be accepted and would not be accepted I disagree. In all cases , even with repeated experiments, the claims are only tentatively accepted. The confirmation by others of Blondlot's N-rays didn't mean they were real, only that stronger evidence would be needed to disprove the conclusions of the earlier observations. Astr…

> But under current medical practices, if a drug trial saw this sort of effectiveness, the trial would be stopped and everyone in the trial offered the treatment. To do otherwise is immoral. As would repeating the same trial.

Then perhaps current medical practices should change. The benefits to those who were previously given the placebo should be balanced against the probability that the observed outcomes may not occur in other circumstances.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#155

Post by munificent »

Most disciplines where correctness is important seem to end up having some adversarial component. It is explicitly how the justice system in the US works [1]. Many software companies have separate QA departments that are deliberately kept at a remove from the engineers to encourage some rivalry between them. Security issues are almost always managed in an adversarial way (though here, you could argue that's because it reflects how the system itself is [mis-]used). Markets are intended to allow fair competition between producers to find an optimal price and product for consumers.

Peer review is supposed to do this, but the fact that peer reviewers are often colleagues leads to collusion, whether intended or not.

Maybe we need a separate body of scientists whose sole job—and whose entire prestige—derives from taking down and retracting bad science.

[1]: https://en.wikipedia.org/wiki/Adversarial_system

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#156

Post by teslaberry »

just be honest, 90% of funded 'social' science is absolute garbage proposals producing expected absolute garbage results.

plenty of 'environmental' and complex system science is just pseudoscience, this includes, especially , 'economics'. which is mostly a confidence game in jargon about complex systems which operate with stupendously unpredictable feedback loops of human behavior. and much of 'economics' is just financial dogma requiring obedience and faith in the current central banking regime as well as in the basic concept of top down control over your money by way of pure authority.

you could defund 90% of these areas and you'd be no worse off, possibly better.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#157

Post by hx87 »

throwaway729 wrote: post #137
JustUhThought wrote: post #123

Earlier quoted context omitted.

Trust but confirm. The entire point of science, and this is not a hyperbole, is that results are reproducible. If the experiment is not reproducible one must take the results on faith. There is no such thing as faith based science. In order to build a shared body of knowledge based on scientific facts, then, results must be repeated. It is how different people can talk about the same thing, without fearing an asymmet…

> The entire point of science, and this is not a hyperbole, is that results are reproducible. No, it's not. The point of science -- its end -- is to understand the natural world. Or to cure diseases. Or, more cynically, to learn how build big bombs and more manipulative adverts. Reproducible results are the means, not the end. I know that seems like hair splitting, but it's important. Epistemological purity can do ju…

The only way we "understand" the natural world is by making verifiable predictions. If those predictions can't be consistently verified, then we don't understand the relevant phenomenon at all, and we haven't learned anything.

> To be concrete about it, I'll always prefer a single (set of) instance(s) of a well-designed and expertly executed experiment over 10 reproductions of a crappy experiment.

I'd take 2-3 repetitions of a moderately well-designed and moderately executed experiments over either. Even the most well-designed and executed experimental protocols can produce spurious results, due to the stochastic nature of the universe.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#158

Post by fragola »

bontoJR wrote: post #11

The big issue right now is funding of replicated research, who wants to fund a research to prove someone else was right? Most of these funds are granted based on potential outcome of the new discovery like: potential business, patents, licenses, etc... not being the first one would probably wipe most of these benefits, cutting down to a small probably getting funded... Now, straight to the point, who's going to pay f…

We can all donate to organizations like the Reproducibility Project: https://osf.io/ezcuj/

But yeah, larger entities (universities, businesses) should also be factoring the cost of reproduction when they commission research.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#159

Post by 8note »

throwaway729 wrote: post #103
return0 wrote: post #79

Earlier quoted context omitted.

We don't need to 'define repetition', we need to foster a culture that a) accepts repetition and b) does not accept something for a fact just because it's in a journal. Right now, (a) is not even acceptable; nobody will publish a replication study. Ofc, LHC is impossible to replicate, but the vast majority of life science studies are. I should think this is mostly needed in life sciences. Other, more 'exact' sciences…

> we need to foster a culture that a) accepts repetition Do we? I don't think we do. I think we need to foster a culture of honesty and rigor. Of good science. Which is decidedly different from fostering a culture of "repetition" for its own sake. Paying for the cost of mountains upon mountains of lab techs and materials that it would require to replicate every study published in a major journal just isn't a good use…

codewise, wouldnt the analog be writing a new set of unit tests for each dependency you pull?

it'd probably make sense to do that, actually, so you can verfiy that the dependency actually fits your use case as time goes on.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#160

Post by jheriko »

wait! we should use the scientific method for science?

its a radical suggestion. sad that it is, but true... ;)

seriously though... you can't have falsifiable results if you don't constantly try to falsify them. then it just becomes a result, which means the conclusion you can draw is close to nothing.... not quite nothing, but exceptionally close. :)

Post reply on HN