Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Should Not Accept Scientific Results That Have Not Been Repeated

nautil.us

Post reply on HN
Print view View original

201–210 of 282 posts

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#201

Post by ramblenode »

I have an alternative proposal: do a study right the first time.

That means:

A) Pre-registering the study design, including the statistical analysis. Otherwise, attaching a big label "Exploratory! Additional confirmation needed!"

B) Properly powering the study. That means gathering a sample large enough that the chances of a false negative aren't just a coin flip.

C) Making the data and analysis (scripts, etc.) publicly available where possible. It's truly astounding that this is not a best practice everywhere.

D) Making the analysis reproducible without black magic. That includes C) as well as a more complete methods section and more automation of the analysis (one can call it automation but I see it more as reproducibility).

Replication of the entire study is great, but it's also inefficient in the case of a perfect replication (the goal). Two identical and independent experiments will have both a higher false negative and false positive rate than a single experiment with twice the sample size. Additionally, it's unclear how to evaluate them in the case of conflicting results (unless one does a proper meta-analysis--but then why not just have a bigger single experiment?).

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#202

Post by physicalist »

ramblenode wrote: post #201

I have an alternative proposal: do a study right the first time. That means: A) Pre-registering the study design, including the statistical analysis. Otherwise, attaching a big label "Exploratory! Additional confirmation needed!" B) Properly powering the study. That means gathering a sample large enough that the chances of a false negative aren't just a coin flip. C) Making the data and analysis (scripts, etc.) publi…

Your proposal is comparable to saying that checks and balances are not needed in a democracy, politicians just need to govern "right". This is about incentivising scientists to do the right thing instead of merely demanding it, like you do.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#203

Post by zouhair »

Obi_Juan_Kenobi wrote: post #192
zouhair wrote: post #81

Earlier quoted context omitted.

You ask that like it is something new. Reproducibility[0] is something at the core of science. The problem these days is that spending time doing replication is not glamourous and will not help you get funds. [0]: https://en.wikipedia.org/wiki/Reproducibility

The problem is worst in fields that don't have reproducibility 'built in' to the field. I do genetics and development, and the main sanity check we have is the distribution of mutant lines. If you say that mutant X does Y, other people are likely going to see that (or not) when they get their hands on it and start poking around. This strength of working with mutants is at the core of the success of molecular biology.…

Add to that the impossibility to publish negative studies. Science really needs to get its shit straight.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#204

Post by zouhair »

analog31 wrote: post #174
zouhair wrote: post #81

Earlier quoted context omitted.

You ask that like it is something new. Reproducibility[0] is something at the core of science. The problem these days is that spending time doing replication is not glamourous and will not help you get funds. [0]: https://en.wikipedia.org/wiki/Reproducibility

I suppose the funding agencies could play a role in this. When you submit a proposal that cites prior work as a basis or motivation for you project, you should be required to show that either 1) The cited works have been reproduced, or 2) you're going to reproduce them.

And also we need to start publishing negative studies way more.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#206

Post by noobermin »

misnome wrote: post #151
welterde wrote: post #83

Earlier quoted context omitted.

ATLAS, CMS, ALICE, LHCb, etc. are all different experiments that just happen to share the same accelerator. They are done using different detectors, designed, built, operated and the data is analyzed by different groups of people. If you follow your logic, that the different experiments using the same accelerator negates the whole thing, to the extreme then doing two experiments on the same planet/solar system/univer…

I think you've got my argument completely backwards - I'm pointing out that if repetition as implied by the article is everything, then results from e.g. the LHC are discounted. Similarly, I am not saying that you don't need to repeat - just that it isn't the be all and end all of what defines 'science'. Supporting this interpretation is that I mentioned DAMA. Nobody is accusing them of not taking care, but nobody re…

There is a philosophical assumption underpining astronomy and cosmology, the so-called "cosmological principle."[0] This is an assumption (albeit not completely unfalsifiable) that physics across the universe at large scales is the same for all observers, that here on earth there should be similar to physics in andromeda. Of course, all scientific experiments done have been done on or near earth, and thus, our interpretation of astronomy on earth is tempered by experiments we do here, for example, we assume that the speed of light here is the same as the speed of light elsewhere.

This is, in principle, a problem. We unfortunately have no way to measure the speed of light in andromeda, unlike what we can here on earth, so we really have no idea if our astronomical models are wrong given the non-constancy of the speed of light in andromeda or elsewhere.

So, yes, in principle not being able to repeat science experiments everywhere in the universe is a problem. However, I think if one thinks a little less broadly, testing newtonian gravity in say, Italy and also in China shows at least across the Earth, the phenomenon is similar. Then, at least one can say, "certainly, gravity is the same in Italy and China, and perhaps across the surface of the Earth." That is a stronger statement than "gravity is this way in Italy." Ordering claims by "scientific goodness", we can say that

   Gravity is the same *across the universe* > gravity is the same across the Earth
                                             > gravity is in Italy
My point here is that even if one can't fulfill the extreme of testing theories everywhere in the universe, one can progressively prove stronger and stronger statements regarding the validity of scientific theories.

Somewhere the LHC stands between "the SM is validated across the universe" and "the SM is validated at one detector at the LHC". Yes, it would be "better" if the Higgs was found at other experiments, but the current situation is "better" than if the Higgs was found at one detector there and not in any other. Repeatability, like everything in science, is not a binary step function but is some continuous function over the domain [0,1].

[0] https://en.wikipedia.org/wiki/Cosmological_principle

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#207

Post by doug1001 »

"repeated" in this context is not incorrect, but i think "replicated" is perhaps a better choice.

That aside, i think repeatability is a much more useful goal (rather than "has been repeated"). For one thing, meaningful replication must be done by someone else; for another, it's difficult and time consuming; the original investigator has no control over whether and when another in the community chooses to attempt replication of his result. What is within their control is an explanation of the methodology they relied on to produce their scientific result in sufficient detail to enable efficient repetition by the relevant community. To me that satisfies the competence threshold; good science isn't infallible science, and attempts to replicate it might fail, but some baseline frequency for ought to be acceptable.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#209

Post by dalke »

jerf wrote: post #180
dalke wrote: post #99

Earlier quoted context omitted.

You want to change the system. I understand that. We have many systems to go on over the last few hundred years of science. We have the pre-war system, primarily funded by private philanthropy. We have the communist system. None of them seem to create the stream of highly replicable studies you want. That may indicate something deep about how people work and how science is really done, and suggest that your admirable…

I tend to agree, actually. In some sense the real solution is take science off its pedestal, as it does not generally deserve to be up there. The 17th-19th centuries were in some sense a fluke of low-hanging fruit, and the science of most of the 20th and the 21st centuries do not deserve to be regarded with the same worshipful gaze, a word I choose carefully. By taking it off its pedestal and subjecting it to a lot m…

"Science" is no longer on a pedestal. The PR campaigns against conclusions for leaded gas, smoking, acid rain, global warming, and vaccine safety, and the scientific development of leaded gas, ozone-depleting CFCs, Agent Orange/dioxin, etc., plus concerns like GMOs and Monsanto, mobile phone safety, plasticizers/hormone disruptors, and more make for a decidedly mixed view of science by the general public.

As you can see from http://www.pewforum.org/2013/07/11/public-esteem-for-militar... , the military, teachers, and medical doctors are on higher pedestals than scientists.

That said, I'm all for the mixed development model.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#210

Post by sidarape »

tnecniv wrote: post #171
sidarape wrote: post #107

Earlier quoted context omitted.

I mean, if a paper gives an algorithm, proves its correctness, and you are convinced by the proofs, then you're done. I don't see how implementing the algorithm gives you more insight. I'm doing my master degree in computational geometry and most people in my lab don't even implement their algorithms. They just know they are correct from their proofs.

Because evaluating algorithms is not always that straight forward. For some algorithms runtime is hugely important, and I don't mean the asymptotic complexity but a hard benchmark of how much it can do in what time span. Stuff like a SLAM algorithm being O(n^2) is nice and all, but to compare it to other SLAM algorithms, I need hard numbers on what it can do in how many milliseconds. Often, I find that authors don't…

Yeah sure but at that point, it's more optimization than algorithm design. Of course, with any algorithm, you always have the hidden constant that you must account for. Also, what I was saying does not apply to the entier CS field. It only applies when you try to design an algorithm for a problem that does not yet have an efficient algorithm. I don't have much experience but I don't think it is really hard to see the cost of the hidden constant in most algorithms.
Post reply on HN