Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Irreproducibility Crisis of Modern Science

nas.org

Post reply on HN
Print view View original

241–250 of 265 posts

Re: The Irreproducibility Crisis of Modern Science

#241

Post by Houshalter »

3JPLW wrote: post #114
lisper wrote: post #68

The site is down so I can't read the original report, but I've read reports on this topic in the past so I'm going to chime in with some "usual suspects" caveats: 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. The best you can hope to do is to reproduce the conditions that matter , but enumerating those has to be part of the theory you are testing, and s…

> The statistical tests currently in widespread use as a criterion for publication in peer-reviewed journals guarantee that at least one result in 20 will be due to chance and not because the hypothesis being tested is actually true. That's wildly overly pessimistic. That would only be the case if scientists just went around, looking at the world, and came up with null-hypotheses willy nilly. That's generally not the…

It's worse than just 5% because p hacking. Every analysis has many parameters, and you can choose the most significant out of thousands of possibilities.

On top of this, the more interesting results are the most surprising. And therefore the least likely to be true a priori. So the chance of an interesting study being true is even lower.

Re: The Irreproducibility Crisis of Modern Science

#242

Post by lmm »

lliamander wrote: post #203
lmm wrote: post #185

Earlier quoted context omitted.

> In those conditions, the goal should be to minimize the consequences of being wrong. We do that by decentralizing decisions to the greater extent possible. In some cases yes. In others this doesn't seem to be the case - e.g. decentralised housing/planning/zoning policy seems to have lead to worse outcomes than more centralised approaches.

Genuinely curious: why do you think this is? I haven't surveyed zoning policies, so I don't know whether I agree with your conclusion, but I'd like to clarify a couple things. First, we must ask ourselves what the goal of optimal zoning policy is. Is it to maximize the number of people who live there? Achieve an ideal rent/wage ratio? Maximize the efficiency of shared infrastructure? Perhaps it is to maximize the des…

> Genuinely curious: why do you think this is?

Because the costs of building new housing are local but the benefits are global, or at least more widely distributed than the costs. If one borough builds denser housing, that borough is the one that has to worry about building work, plumbing, schools and all the rest of it for those residents, while businesses from all its neighbours benefit from more customers and cheaper employees. For a starker example of the same effect think about taxation and public services: if we relied on local taxation being spent locally then rich areas would have low tax rates and high-quality public services (because even at low rates they can still raise a lot of money) while poor areas would have both higher taxes and worse services.

Localised government is rather like privatisation: it works well when the costs and benefits are on the same scale, but rather less well when there are externalities.

Re: The Irreproducibility Crisis of Modern Science

#243

Post by geofft »

gehwartzen wrote: post #236
geofft wrote: post #113

Earlier quoted context omitted.

I don't see why you would want to solve this with a researcher basic income (and overhead/bureaucracy of figuring out who is a researcher) and not the simpler solution of an actual universal basic income.

I don't think a $10-20k/yr UBI is anywhere close to the actual cost of allowing individuals to conduct meaningful scientific studies.

Grad students seem to make it work, somehow.

Re: The Irreproducibility Crisis of Modern Science

#244

Post by jrochkind1 »

DanielBMarkham wrote: post #211
jrochkind1 wrote: post #201

Earlier quoted context omitted.

Good catch thanks! Wikipedia says "The National Association of Scholars (NAS) is an American non-profit politically conservative advocacy group, with a particular interest in education" Thanks for the link to the PNAS -- looks like a whole special issue, not just an article, in fact. Great. I'm gonna ignore the fake-NAS one, and read the real-PNAS one. I wonder what National Association of Scholars' motivation is her…

Isn't the entire purpose of reproducability is to eliminate such questions from discussions about science?

To eliminate what such questions?

To eliminate any questions about social context or bias or agenda? Totally not possible, social context is an inherent part of science, as in all human endeavors.

I think the purpose of reproducibility is to well, make science _possible_, it is the fundamental bedrock of science -- to determine what and how we can make reliable predictions about the world. It doesn't mean bias and effects of social context no longer exists though. Science exists anyway, in a social context -- as evidenced by the past couple hundred years!

In general, I don't think science exists to eliminate _any_ questions, science exists to ask questions. Eliminating questions may be a goal of ideology or religion, but not science. (and in the real world, they certainly influence each other. science is a goal to strive for, not a fait accompli. Scientific understandings of the world are always changing; science is an approach to understanding the world, not a set of settled conclusions).

Re: The Irreproducibility Crisis of Modern Science

#245

Post by merpnderp »

emodendroket wrote: post #171
merpnderp wrote: post #54

Earlier quoted context omitted.

The same Krugman who pronounced the markets would never recover from Trump's election or that since we didn't pass a large enough stimulus we were doomed to low GDP growth rates forever?

Paul Krugman was wrong on different issues, thereby completely proving everyone who agreed with him on this other issue wrong. Got them.

No he's wrong because he supported a policy which saw the US debt double in 8 years, and the deficit go from 52% of GDP to 77% of GDP, and resulted in the weakest recovery in US history. And all the people who agreed with him in their widely accept science, were also wrong.

Re: The Irreproducibility Crisis of Modern Science

#246

Post by gehwartzen »

geofft wrote: post #243
gehwartzen wrote: post #236

Earlier quoted context omitted.

I don't think a $10-20k/yr UBI is anywhere close to the actual cost of allowing individuals to conduct meaningful scientific studies.

Grad students seem to make it work, somehow.

Fair point and even I managed to live off of and conduct research on 25k/yr in grad school. Though much of that cost was also offset by access to facilities, labs, equipment, and supplies. Would love to see more public access to these sorts of resources.

Re: The Irreproducibility Crisis of Modern Science

#247

Post by air7 »

A good solution would be to simply reduce the p-value threshold.

The current "standard" p-value in many fields, of 5%, is arguably too high. consider throwing a double six in backgammon has probability of 3%. That means that, on some level, throwing 6-6 would be a valid scientific "proof" of ESP. (p-valueA high p-value is basically externalizing part of the research cost onto other scientists and in the process creates a lot of false-positive "noise".

Re: The Irreproducibility Crisis of Modern Science

#248

Post by lisper »

buvanshak wrote: post #238
lisper wrote: post #237

Earlier quoted context omitted.

Yes, all that is true. What does that have to do with reproducibility? You're talking about doing a lot of different experiments to test a single theory. Reproducibility is about doing the same experiment over again to see if you get the same results as the first time you did it, i.e. reproduce the results.

> What does that have to do with reproducibility? That is exactly what I wanted to ask you originally when you cited the celestial mechanics example. It is not relevant in cases we are deriving a theory underlying the behavior , rather than axiomizing the very specific behavior itself...

I don't understand the difference between "deriving a theory underlying the behavior" and "axiomizing [sic] the very specific behavior itself."

Re: The Irreproducibility Crisis of Modern Science

#249

Post by jrochkind1 »

jrochkind1 wrote: post #244
DanielBMarkham wrote: post #211

Earlier quoted context omitted.

Isn't the entire purpose of reproducability is to eliminate such questions from discussions about science?

To eliminate what such questions? To eliminate any questions about social context or bias or agenda? Totally not possible, social context is an inherent part of science, as in all human endeavors. I think the purpose of reproducibility is to well, make science _possible_, it is the fundamental bedrock of science -- to determine what and how we can make reliable predictions about the world. It doesn't mean bias and ef…

Also, if we're looking for the social context that leads to the reproducibility crisis, I think it's a lot more likely to be about academic career incentives (you _gotta_ publish things, a lot of things, that all seem like important and socially useful results, none of which are negative results -- when the actual practice of science means negative results are an expected and useful thing, not indication you are a bad scientist), than the political ideologies/biases of researchers.

Maybe I'm wrong. There are probably ways to investigate it. This essay isn't one of them.

Post reply on HN