Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Irreproducibility Crisis of Modern Science

nas.org

Post reply on HN
Print view View original

111–120 of 265 posts

Re: The Irreproducibility Crisis of Modern Science

#111

Post by sctb »

jnordwick wrote: post #85
Afforess wrote: post #28

Earlier quoted context omitted.

Ad hominem has never been a valid reason to disbelieve something. I quite enjoyed reading the report - it includes XKCD references and everything.

That you're being voted down shows that HN has really changed. (I blame all the overtly political articles that I believe have changed the tenor of the site, and politics has infected every article now. Thanks @dang!)

If you won't stop posting this off-topic meta noise that we've asked you many times before to cut out, we'll ban the account. Post according to the guidelines or not at all. Fulminate elsewhere.

https://news.ycombinator.com/newsguidelines.html

Re: The Irreproducibility Crisis of Modern Science

#112

Post by chrisfosterelli »

Erlich_Bachman wrote: post #79
lisper wrote: post #68

The site is down so I can't read the original report, but I've read reports on this topic in the past so I'm going to chime in with some "usual suspects" caveats: 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. The best you can hope to do is to reproduce the conditions that matter , but enumerating those has to be part of the theory you are testing, and s…

4.) You mean that papers are published with P When did this become a "standard"? Most papers I have seen in respectable journals and other sources would not strive for such a high Pvalue, it would be much lower like 0.001.

> When did this become a "standard"?

I think this depends on the field. In some areas of neuroscience for example, brain data can have extremely high variability and is very expensive to collect a lot of, so p<0.05 is not uncommon.

Re: The Irreproducibility Crisis of Modern Science

#113

Post by geofft »

TangoTrotFox wrote: post #99
danharaj wrote: post #76

Lets take a page from Marx. Science is many things, in particular a relationship between capital and labor. The scientific method is a wonderful idea, but it is subordinate to the economic forces that underlie scientific activity. Look at the conflicts and contradictions between those doing science (labor) and those deciding the science to be done (capital), and that is the ultimate source of these crises. The execut…

I think this is pretty self evident, but the issue is that let's say you have some system where economic forces are removed. Essentially a researcher basic income in one scenario. This would suddenly massively incentivize people towards this direction since it's basically a career path that guarantees a stable livelihood, which is something that's extremely rare today. Well you need to ensure there's nobody just comp…

I don't see why you would want to solve this with a researcher basic income (and overhead/bureaucracy of figuring out who is a researcher) and not the simpler solution of an actual universal basic income.

Re: The Irreproducibility Crisis of Modern Science

#114

Post by 3JPLW »

lisper wrote: post #68

The site is down so I can't read the original report, but I've read reports on this topic in the past so I'm going to chime in with some "usual suspects" caveats: 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. The best you can hope to do is to reproduce the conditions that matter , but enumerating those has to be part of the theory you are testing, and s…

> The statistical tests currently in widespread use as a criterion for publication in peer-reviewed journals guarantee that at least one result in 20 will be due to chance and not because the hypothesis being tested is actually true.

That's wildly overly pessimistic. That would only be the case if scientists just went around, looking at the world, and came up with null-hypotheses willy nilly. That's generally not the case. There is often a _reason_ for conducting the test and a plausible cause of action. There are two possible reasons for a significant result:

* The null hypothesis is correct but they got "unlucky" data (5% chance or p% chance)

* There is a real effect (and the null hypothesis is actually wrong)

This becomes more problematic in reproducibility tests, though, since that biases my prior towards a correct null hypothesis and now you must be very careful about pre-registration and the numbers of folks worldwide that are trying to reproduce a given experiment.

Re: The Irreproducibility Crisis of Modern Science

#115

Post by AnimalMuppet »

TangoTrotFox wrote: post #109
buvanshak wrote: post #86

Earlier quoted context omitted.

>scientists criticizing how the scientific process is currently conducted for the purposes of improving the scientific endeavor. I think what happening here is a bit more serious. They are showing a widespread crisis. It is not just some minor feedback to improve the process. >It does not in itself show that science is inherently untrustworthy. I think when statistics is involved, the results are inherently untrustwo…

> "I think when statistics is involved, the results are inherently untrustworthy. This is not really surprising because there is a whole bunch of ways these studies that involve statistics could go wrong. And we are still finding new ways on how this could go wrong." Another very real issue here is that malicious use of statistics can be used to show nearly anything in ways that can be extremely difficult to detect,…

Feynmann said, "The first rule is that you must not fool yourself. And you are the easiest person to fool."

There can be malice, sure. But there can also be desire to believe. And, hey, here's a statistical analysis that shows what the investigator is already biased to believe anyway...

Re: The Irreproducibility Crisis of Modern Science

#116

Post by TangoTrotFox »

haZard_OS wrote: post #107
buvanshak wrote: post #86

Earlier quoted context omitted.

>scientists criticizing how the scientific process is currently conducted for the purposes of improving the scientific endeavor. I think what happening here is a bit more serious. They are showing a widespread crisis. It is not just some minor feedback to improve the process. >It does not in itself show that science is inherently untrustworthy. I think when statistics is involved, the results are inherently untrustwo…

>I think when statistics is involved, the results are inherently untrustworthy. Ummm...are you kidding? Statistically vetted results are inherently UNCERTAIN, but how could they possibly be inherently untrustworthy? Even if a mechanistic effect is observed, its relationship to a particular cause or influence is only established statistically. In fact, the very observation is often performed u Dee the umbrella of stat…

I cannot speak for the person you are responding to, but in my [agreement] of his critique of statistics, I am implicitly speaking of social statistics. I think there is a vast difference in e.g. a statistical modeling of the behavior of electrons and e.g. the statistical modeling of some sort of human behavior.

Re: The Irreproducibility Crisis of Modern Science

#117

Post by jxub »

geofft wrote: post #113
TangoTrotFox wrote: post #99

Earlier quoted context omitted.

I think this is pretty self evident, but the issue is that let's say you have some system where economic forces are removed. Essentially a researcher basic income in one scenario. This would suddenly massively incentivize people towards this direction since it's basically a career path that guarantees a stable livelihood, which is something that's extremely rare today. Well you need to ensure there's nobody just comp…

I don't see why you would want to solve this with a researcher basic income (and overhead/bureaucracy of figuring out who is a researcher) and not the simpler solution of an actual universal basic income.

Simpler but still more expensive by far, and I suppose the ROI would be far smaller.

Re: The Irreproducibility Crisis of Modern Science

#118

Post by AstralStorm »

merpnderp wrote: post #38
lmm wrote: post #12

Earlier quoted context omitted.

If we don't use science to make decisions, what are we supposed to use? Gut instinct? Of course science is imperfect, but it tends towards truth over time, whereas dogma and populism have no way of correcting mistakes that isn't just as likely to introduce new ones.

How long ago was it that science was controverting centuries of populist knowledge that carbs are what made you fat, not eating fat? Talk about science creating a mind boggling ongoing health crisis by telling people to limit their fat intake and replace it with cereals and bread.

If by science you mean policy of NHS?

Current scientific consensus is that it is caloric surplus - confounded by a bunch of poorly researched factors. For weight gain source does not seem to matter too much. This does not say anything about dietary specifics or how they affect intake and metabolism.

Technically nutrition science does not provide strong support for such statements as this carb vs fat one based on interventional studies.

Epidemiology alone does not tell you "therefore this change will be beneficial"...

Re: The Irreproducibility Crisis of Modern Science

#119

Post by hpcjoe »

lisper wrote: post #94
Erlich_Bachman wrote: post #79

Earlier quoted context omitted.

4.) You mean that papers are published with P When did this become a "standard"? Most papers I have seen in respectable journals and other sources would not strive for such a high Pvalue, it would be much lower like 0.001.

> You mean that papers are published with P I'm guessing you meant to say that no papers are published with P at least one paper in 20 will be "wrong". Then add self-suppression of negative results and the actual percentage of "wrong" papers becomes higher, potentially much higher if real positive results are rare. > When did this become a "standard"? I have no idea. You'll have a ask a historian.

Part of the mistake here is in interpreting the p value as being a measure of anything really meaningful. PIt is not, in any way a measure of reality. It is a guess at how accurately you've modeled the system, how accurate your priors are, and so forth.

Re: The Irreproducibility Crisis of Modern Science

#120

Post by knodi123 »

geofft wrote: post #113
TangoTrotFox wrote: post #99

Earlier quoted context omitted.

I think this is pretty self evident, but the issue is that let's say you have some system where economic forces are removed. Essentially a researcher basic income in one scenario. This would suddenly massively incentivize people towards this direction since it's basically a career path that guarantees a stable livelihood, which is something that's extremely rare today. Well you need to ensure there's nobody just comp…

I don't see why you would want to solve this with a researcher basic income (and overhead/bureaucracy of figuring out who is a researcher) and not the simpler solution of an actual universal basic income.

> and not the simpler solution of an actual universal basic income

Eliminating fossil fuel use is also simple. But there's another dimension to this problem, called "political viability", which I feel you're overlooking.

Post reply on HN