Live data from Hacker News

We Should Not Accept Scientific Results That Have Not Been Repeated

nautil.us

81–90 of 282 posts

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#81
post #59

Define repetition. It's not as simple as that, for all sciences - once again an article on repeatability seems to have focused on medicinal drug research (it's usually that or psychology), and labelled the entire "Scientific community" as 'rampant' with " statistical, technical, and psychological biases". How about, Physics? The LHC has only been built once - it is the only accelerator we have that has seen the Higgs…

You ask that like it is something new. Reproducibility[0] is something at the core of science.

The problem these days is that spending time doing replication is not glamourous and will not help you get funds.

[0]: https://en.wikipedia.org/wiki/Reproducibility

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#82
post #72
post #61

Earlier quoted context omitted.

> So, $1 of public funding triggers public code release? Or is there a threshold? Yes, 1$, no threshold necessary. For projects where the cost of publication of data is not worth the dollar, people will just not accept the dollar. (That's where a 'natural threshold' comes in.)

So, if I have a summer intern who is paid by the government, and helps on my otherwise non-government-funded project, then is that enough to trigger the requirement? Or, if I use a government facility, like a supercomputing center, then that also trigger the release requirement? Even if no money changes hands? What about a government network? It sounds very tricky. If I get a government grant to work on a project, an…

No it's very simple. If you want "the people" to pay for you to do things at all, you owe them all of the things or you don't get their money. Your objections make it seem complicated but this feels like an effort to muddy fairly clear waters.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#83
post #59

Define repetition. It's not as simple as that, for all sciences - once again an article on repeatability seems to have focused on medicinal drug research (it's usually that or psychology), and labelled the entire "Scientific community" as 'rampant' with " statistical, technical, and psychological biases". How about, Physics? The LHC has only been built once - it is the only accelerator we have that has seen the Higgs…

ATLAS, CMS, ALICE, LHCb, etc. are all different experiments that just happen to share the same accelerator. They are done using different detectors, designed, built, operated and the data is analyzed by different groups of people.

If you follow your logic, that the different experiments using the same accelerator negates the whole thing, to the extreme then doing two experiments on the same planet/solar system/universe won't be enough..

And I don't buy your inverse argument that one (good) experiment is good enough either. It is very difficult to tell from outside the group if the experiment is actually good or not (although you probably can tell if it's bad). Screw-ups can happen no matter how many people look at the data if there is some flaw in the experimental setup - only way to make really sure is to use different experiments to measure the same thing.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#84
We shouldn't "accept" or "reject" results at all.

It's not a binary option. One poor experiment might give us some evidence something is true. A single well reviewed experiment gives us more confidence. Repeating the results similarly does. As does the reputation of the person conducting the experiment and the way in which it was conducted.

It's not a binary thing where we decide something is accepted or rejected, we gather evidence and treat it accordingly.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#85
post #59

Define repetition. It's not as simple as that, for all sciences - once again an article on repeatability seems to have focused on medicinal drug research (it's usually that or psychology), and labelled the entire "Scientific community" as 'rampant' with " statistical, technical, and psychological biases". How about, Physics? The LHC has only been built once - it is the only accelerator we have that has seen the Higgs…

I think the difference is that we are able to test and understand/verify the operation of the LHC at multiple levels. It's not like they just built what was on paper, flicked the ON switch, and assumed all the data that came out was correct. The components were individually tested and collisions at energies accessible by other supercolliders were done so we could compare to other results. Unfortunately for medical dr…

> but that these fields by their nature make clean hypothesis testing vastly more difficult and p-hacking and statistical trickery (intentional or otherwise) harder to sift out.

How certain are we that its 'the nature' of these fields rather than political choices that were solidified long ago? There could be much larger samples and six-sigma confidence in life sciences if the grants were allocated differently. I know this is happening, for example, in neuroscience where the (private) Allen Institute is creating significantly more rigorous (and useful) datasets compared to the bulk of studies, because they are funding their studies differently.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#86
post #59

Define repetition. It's not as simple as that, for all sciences - once again an article on repeatability seems to have focused on medicinal drug research (it's usually that or psychology), and labelled the entire "Scientific community" as 'rampant' with " statistical, technical, and psychological biases". How about, Physics? The LHC has only been built once - it is the only accelerator we have that has seen the Higgs…

Thinking about physics, the LHC experiment benefits from not being a completely isolated study. Instead, its results are multiple pieces of a larger jigsaw puzzle or web of interconnected results that include both a quantitative predictive theory and other experiments that test other predictions of the same theory. And the Higgs was not the only thing that could be tested at the LHC. For instance, it probably replicated a huge slew of prior studies during construction and testing of the system, from already-known particles all the way down to the calibration of voltmeters.

In physics, some theories have been tested and interconnected to such a degree, that if an experiment conflicts with theory, it's reasonable to suspect the experiment rather than discard the theory. That's what happened with that apparent faster-than-light travel of neutrinos captured in Italy. It was something like a partially unplugged cable. Once corrections were made for the cable, the theory snapped right back into place. Those theories can be trusted as akin to tools in day to day physics. For my humble graduate experiment, refutation of any major law would have led me to fix the experiment and try again. I simply assumed things like conservation of charge. Such laws probably include electrodynamics, gravitation, quantum mechanics, thermodynamics, and Darwinian evolution.

And this is a common feature of major studies in physics, chemistry, evolutionary biology, and other sciences as well.

Where we may run into trouble is in branches of sciences that don't have over-arching theories or that web of connected results. At the other end of the scale are areas of sciences where the results are mainly a database of observed statistical correlations, with little or no apparent progress towards a general theory. When someone publishes a surprising result, there is no reason to say that the experiment must have been done wrong. You just add it to the pile. The best that can be hoped for is that some kind of meta-analysis will demonstrate an inconsistency among multiple studies. Those fields don't have the "hard" theories that can be used as tools to test experimental results.

To be fair, there's another situation, where you have not one, but zero, reproducible results. That's the state of affairs in the search to unify gravity and quantum mechanics.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#87
post #72

Earlier quoted context omitted.

So, if I have a summer intern who is paid by the government, and helps on my otherwise non-government-funded project, then is that enough to trigger the requirement? Or, if I use a government facility, like a supercomputing center, then that also trigger the release requirement? Even if no money changes hands? What about a government network? It sounds very tricky. If I get a government grant to work on a project, an…

No it's very simple. If you want "the people" to pay for you to do things at all, you owe them all of the things or you don't get their money. Your objections make it seem complicated but this feels like an effort to muddy fairly clear waters.

I'm pointing out that it's impossible to work this way. People don't like it. The accounting systems aren't set up for this. The entire post-war research system isn't structured this way.

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#88
post #6

Maybe conferences should have a "reproducibility" track for that purpose? Also, I don't know about other fields, but I'm pretty sure that in CS, if you just took a paper and tried to reproduce the results, you'll get rejected on the ground that you offer no original contribution; no original contribution => no publication => no funding.

I don't think reproducibility should necessarily mean literally repeating the same study over again. Whether or not a paper is fully correct is less important that the further work it stimulates or informs-- either via more papers (impact factor) or via APPLICATIONS of the research. In the example you're giving, to get accepted, the author doesn't need to do much more than extend the research in some small way, compa…

"I don't think reproducibility should necessarily mean literally repeating the same study over again."

Yes, it should. We have too many demonstrated instances of studies built on other studies that turned out to be flawed, but the second study, rather than showing the first one was flawed, was rationalized and massaged until it conformed to the first study because the first study already had the imprimatur of peer-reviewed correctness on it.

Only someone replicating the original study directly (give or take sample size or duration or other simple such changes) will have the guts and moral authority to stand up and say "I can't replicate this. The original study may be wrong."

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#89
Lots of scientific results are repeated but not published. If it doesn't work then people just move on. The problem is journals. There is no way to publish your attempts to repeat an experiment, unless you put it into another paper.

The other issue, especially in the life sciences, is inaquedate statistical input. If someone performs an underpowered, confounded experiment and gets a positive result, then someone else performs the same underpowered confounded experiment and gets a negative result, what have we learned except that the experiment is underpowered?

Re: We Should Not Accept Scientific Results That Have Not Been Repeated

#90
post #87

Earlier quoted context omitted.

No it's very simple. If you want "the people" to pay for you to do things at all, you owe them all of the things or you don't get their money. Your objections make it seem complicated but this feels like an effort to muddy fairly clear waters.

I'm pointing out that it's impossible to work this way. People don't like it. The accounting systems aren't set up for this. The entire post-war research system isn't structured this way.

The entire post-war research system is set up to create endless streams of non-replicable studies based on incorrectly-applied statistics all the while convincing itself that it's doing something useful while wasting billions upon billions of dollars.

I'm not to impressed by "well, that's not how it works right now". The whole problem is "how it works right now". That's what we're discussing, the need for it to not work that way.

Post reply on HN