The Irreproducibility Crisis of Modern Science

> 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment.

This is why you include the error in your results. We don't care if your two experiments result in 100% the same results. We care about if your results line up within the error of your experiment.

> 2. Even a completely non-reproducible result can be scientifically significant. For example, celestial events are almost never reproducible. Our understanding of celestial mechanics nonetheless rests on solid science.

This is why we share data. That is in essence our observation. A celestial event might be caught by only one instrument, but several people can develop several different models. We wait and watch for similar events though, to check if models are consistent (within error).

> 3. The end-product of science is not truth, it is explanations of observations.

I just wanted to repeat this because it can never be stated enough.

> 4. The statistical tests currently in widespread use as a criterion for publication in peer-reviewed journals guarantee that at least one result in 20 will be due to chance and not because the hypothesis being tested is actually true.

While I don't like p values, because of hacking, (especially 0.05) that's not how stats work. Flipping 10 heads doesn't guarantee 5 tails, not even 1. I wouldn't use as strong as a word as guarantee.

But this is also an argument FOR reproducing. If multiple experiments are consistent with one another (within error) than that strengthens the argument.

TLDR: More brains that look at a problem helps solve the problem.

I will add my own statements about the reproducability problem in science. One is because there is less funding for it. Reproducing an experiment isn't sexy. Another problem just stems from that data isn't always open. It is hard to review work if you don't know everything about the experiment. Data can even have simple mistakes that just weren't caught. But it is also embarrassing to share data.

> How long ago was it that science was controverting centuries of populist knowledge that carbs are what made you fat, not eating fat?

I've looked into that and all I can say is the history is disputed - some people say there really was a nutritionist consensus, others that this was a popular misunderstanding based on overzealous popularisation of a small number of papers (similar to the myth that the consensus among climate scientists in the '70s predicted an ice age) and that working scientists never thought this.

Either way it's a failing of the process, but the appropriate response is quite different; in the first case all we can say is mistakes do happen, in the second we need to get better at ensuring that the real scientific consensus gets translated into appropriate policy actions.

Marx!? Marx can teach us very little about economics and almost nothing about science. Science is NOT “a relationship between capital and labor”.

If you see everything through a such a strong lens, you see very little.

> I certainly believe that some individuals will be better informed and better able to translate that information into the best course of action for themselves given their goals than governments, especially "governments" as represented by a patchwork of laws and regulations the impact a complex thing like individual health.

Some individuals, sure. Most individuals, probably not.

I agree that government policy has inertia, and that overconfidence on uncertain issues can lead to imposing well-intentioned policy that ends up being harmful. But it's equally possible to make the opposite mistake, and hold off introducing policy changes for too long because of perceived uncertainty. E.g. car seatbelt laws and tobacco advertising laws should have been introduced earlier than they were. I don't think there's any general principle we can draw here; some government policymakers could do with more humility, others could do with more confidence, but all any one of them can do "from the inside" is try to figure out how strong the evidence actually is on a given issue, and use their own best judgement in determining whether there's a clear enough scientific/medical consensus to warrant policy changes, or not.

Never mind the massive debate that occurs within the scientific community itself on some new or controversial claims. When the press gets its hands on the views of one dissenter and one proponent, it makes it seem like there are only sides A and B. This is not as simplistic as politics, where politicians form into immutable groups Left vs Right. We are dealing with numerous camps, and within each scientific camp, there are numerous arguments made for/against an issue.

Take climate change, no doubt a controversial issue. To say it is "controversial" in the political sense would mean Left and Right (in the USA alone) have vociferously different stances on the issue. However, if viewed as a scientific controversy, we are now talking about detailed methodological concerns, like methods of data collection, analysis, kinds of statistical bias, and subtle changes in arbitrary parameters. Any scientist can tweak this or that in their model to make it conform more easily to their preconceived notions about climate change. Unfortunately, we also have some oil industry shills out there who got trotted out as an equally weighted side B to the side A of the dozens of scientists who would generally disagree. Then, you also have anti-science proponents who use the legitimate self-criticism of scientists to attack science as a whole.

It's a sad state of affairs. Science should be reported in the press, but it should also be reported much better than it is. In the USA in particular, STEM education is lagging behind: the average person can't delineate the good science coverage from the bad, and we have ridiculous notions and conspiracies that fail to become filtered out (anti vax, climate change deniers, flat earthers, moon landing was faked, etc.)

For me, reproducibility is one problem in a broader ecosystem of scientific problems, including science education generally, as well as misuse of statistics, and a saddening drive for incremental results at the expense of more broad-based thinking which might lead to fundamental breakthroughs. Our education systems must be reformed to deal with these problems, that's the only way out that I see.