Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Irreproducibility Crisis of Modern Science

nas.org

Post reply on HN
Print view View original

201–210 of 265 posts

Re: The Irreproducibility Crisis of Modern Science

#201

Post by jrochkind1 »

Thrymr wrote: post #198

To be clear, the "NAS" (nas.org) that published this study is the National Association of Scholars [0], a political group, not the National Academy of Sciences (nasonline.org) [1], a nongovernmental organization that consists of scientists elected by their peers to provide independent scientific advice to the US government. There was in fact a study published recently in PNAS, the Proceedings of the National Academy…

Good catch thanks! Wikipedia says "The National Association of Scholars (NAS) is an American non-profit politically conservative advocacy group, with a particular interest in education"

Thanks for the link to the PNAS -- looks like a whole special issue, not just an article, in fact. Great. I'm gonna ignore the fake-NAS one, and read the real-PNAS one.

I wonder what National Association of Scholars' motivation is here exactly.

Re: The Irreproducibility Crisis of Modern Science

#202

Post by lotsofpulp »

foldr wrote: post #195
lotsofpulp wrote: post #177

Earlier quoted context omitted.

As I wrote above, studies about people, which the aforementioned Nazi stuff would fall into. I don't have a specific study in mind, but I do know that the further you stray from math/physics/chemistry, creating metrics and isolating variables is very, very difficult. I don't need a specific study to know that. Any sociology/economic/psychology research should be taken with a grain of salt.

>Any sociology/economic/psychology research should be taken with a grain of salt. So how is that not an example of "science going wrong"? Unless you are proposing that science is right by definition, or that governments should ignore most of the branches of science that are actually relevant to government. There are plenty of branches of physics that are at least as speculative as sociology, economics and psychology…

It's not going wrong, unless it's making conclusive statements that are beyond the strengths of the study. That's why I say grain of salt. I understand what you're saying, but my point is that it's not science or the scientific method that results in undesirable outcomes, such as the Nazi example, but just a lack of any other good option.

Governments (and people) should evaluate the merits of the information they receive. Properly conducted and repeated experiments with noted mechanisms of actions (e.g. vaccines) should have more weight than studies about trickle down wealth or whatever. Obviously, decisions have to be made, and some will be wrong unfortunately.

Re: The Irreproducibility Crisis of Modern Science

#203

Post by lliamander »

lmm wrote: post #185
lliamander wrote: post #126

Earlier quoted context omitted.

> If we don't use science to make decisions, what are we supposed to use? Gut instinct? I thought the parent comment made it clear: the alternative we are supposed to use is "democracy". That is, the people who are ruled should determine the rules that are imposed on them. Now, democracy does not preclude the use of science to make decisions. The voters may rely on science to inform their decisions (when the science…

> In those conditions, the goal should be to minimize the consequences of being wrong. We do that by decentralizing decisions to the greater extent possible. In some cases yes. In others this doesn't seem to be the case - e.g. decentralised housing/planning/zoning policy seems to have lead to worse outcomes than more centralised approaches.

Genuinely curious: why do you think this is?

I haven't surveyed zoning policies, so I don't know whether I agree with your conclusion, but I'd like to clarify a couple things.

First, we must ask ourselves what the goal of optimal zoning policy is. Is it to maximize the number of people who live there? Achieve an ideal rent/wage ratio? Maximize the efficiency of shared infrastructure?

Perhaps it is to maximize the desirability of living in an area. If that is the case, then if people are complaining about the house prices that may mean you're doing something right.

In any case, assuming we settle on a shared metric (which may involve a combination of these various factors), we need to distinguish between comparing the average case and the worst case performance of centralized vs. decentralized zoning policy.

I would say that my claim is that decentralized zoning policy will have a significantly better worst-case than centralized zoning policy, and that crafting a good zoning policy is enough of a gamble that I'd rather settle for a less-than-ideal decentralized policy than risk a horrendous centralized one.

Now, to tie back to the original debate of scientific expertise in particular: would adding a group of economists and social scientists to the central planning committee tilt the situation in favor of central planning? I very much doubt it.

"At the core […] is the idea that people should design for themselves their own houses, streets and communities. This idea […] comes simply from the observation that most of the wonderful places of the world were not made by architects but by the people."

— Christopher Alexander et al., A Pattern Language, front bookflap[1]

[1] https://en.wikipedia.org/wiki/A_Pattern_Language

Re: The Irreproducibility Crisis of Modern Science

#204

Post by dang »

spikels wrote: post #186
danharaj wrote: post #76

Lets take a page from Marx. Science is many things, in particular a relationship between capital and labor. The scientific method is a wonderful idea, but it is subordinate to the economic forces that underlie scientific activity. Look at the conflicts and contradictions between those doing science (labor) and those deciding the science to be done (capital), and that is the ultimate source of these crises. The execut…

Marx!? Marx can teach us very little about economics and almost nothing about science. Science is NOT “a relationship between capital and labor”. If you see everything through a such a strong lens, you see very little.

Please don't take HN threads on generic ideological tangents, which never lead anywhere new. I know the GP pointed there, but that's no reason to follow the pointer.

https://news.ycombinator.com/newsguidelines.html

Re: The Irreproducibility Crisis of Modern Science

#205

Post by [deleted] »

throwawaymarx wrote: post #102
TangoTrotFox wrote: post #99

Earlier quoted context omitted.

I think this is pretty self evident, but the issue is that let's say you have some system where economic forces are removed. Essentially a researcher basic income in one scenario. This would suddenly massively incentivize people towards this direction since it's basically a career path that guarantees a stable livelihood, which is something that's extremely rare today. Well you need to ensure there's nobody just comp…

Small correction. A basic income is guaranteed no matter what; there’s no means testing or anything like that. You can’t game the system because there’s nothing to game: you get the money regardless. You seem to be describing the system we have now, where you’re paid for results.

[deleted]

Re: The Irreproducibility Crisis of Modern Science

#206

Post by iguy »

emodendroket wrote: post #30
scblock wrote: post #21

Earlier quoted context omitted.

So, 'science' not science? Got it. That was a pretty quick Godwining, by the way.

Those views really were thought to be rational and scientific at the time they were promulgated, though, and Nazism was only their most extreme expression.

The problem with the Nazis is that they were genocidal maniacs, not that they p-hacked their scientific papers. Genocidal maniacs have spouted many creeds, some of which sound much nicer (like brotherly love & eternal life... or equality of mankind & ownership of the means of production) and they are still evil. Because mass murder is wrong. Full stop.

I think it's really dangerous to tie the wrongness of the supposedly scientific facts they spouted to the wrongness of their deeds. Dangerous because if it turns out they were right about some point, then you look like an idiot and have to change your tune. And dangerous because it warps subsequent science, to avoid such embarrassment.

They believed in breeding a master race. In response to this, the acceptable opinion in the West ever since seems to have been that this is impossible, that humans are unique among creatures on this planet in that they are immune to the pressures of selective breeding. We can double the size of our chickens (and breed tame foxes, and smart dogs) but humans are apparently immutable. This is creationist nonsense, and it becomes more and more obviously nonsense every year. We need not have painted ourselves into such a stupid corner.

Re: The Irreproducibility Crisis of Modern Science

#207

Post by eli_gottlieb »

lisper wrote: post #133
eli_gottlieb wrote: post #125

Earlier quoted context omitted.

>The end-product of science is not truth, it is explanations of observations. That sounds a whole lot like truth to me, or at least, like any sane construal of truth.

That depends on what you mean by "truth". Newtonian mechanics says that gravity is pulling you down towards the surface of the earth. Is that "true"? It has tremendous explanatory power, but I would argue that it is not actually true. The truth is (as far as we can tell at the moment) that the surface of the earth is pushing you up and accelerating you through curved space-time.

>The truth is (as far as we can tell at the moment) that the surface of the earth is pushing you up and accelerating you through curved space-time.

And in fact, General Relativity is the best explanation science has given us at the moment, which seems to fit what I said.

Re: The Irreproducibility Crisis of Modern Science

#208

Post by iguy »

lliamander wrote: post #126
lmm wrote: post #12

Earlier quoted context omitted.

If we don't use science to make decisions, what are we supposed to use? Gut instinct? Of course science is imperfect, but it tends towards truth over time, whereas dogma and populism have no way of correcting mistakes that isn't just as likely to introduce new ones.

> If we don't use science to make decisions, what are we supposed to use? Gut instinct? I thought the parent comment made it clear: the alternative we are supposed to use is "democracy". That is, the people who are ruled should determine the rules that are imposed on them. Now, democracy does not preclude the use of science to make decisions. The voters may rely on science to inform their decisions (when the science…

too causally dense for us to ... make accurate predictions.

And even if we had perfect predictions, we would still have to choose what to do.

For example, if we had a perfect economics, we could predict exactly how many people would be made unemployed compared to how many made how much richer by raising the minimum wage (or trade barriers, or taxes). How should we weight these?

I think it's easy to greatly overstate how useful science is to policy-making. There are narrow areas where they should (and do) call a specialist in. But the big debates are not over factual, empirical, matters.

Re: The Irreproducibility Crisis of Modern Science

#209

Post by iguy »

lotsofpulp wrote: post #177
foldr wrote: post #146

Earlier quoted context omitted.

Which studies are you referring to? If you don't have anything specific in mind, then you can't really judge the quality or compare it to the quality of comparable research today.

As I wrote above, studies about people, which the aforementioned Nazi stuff would fall into. I don't have a specific study in mind, but I do know that the further you stray from math/physics/chemistry, creating metrics and isolating variables is very, very difficult. I don't need a specific study to know that. Any sociology/economic/psychology research should be taken with a grain of salt.

It would be helpful if you would name specific examples where you think they were wrong. I mean wrong on empirical questions, not morally wrong.

Just stating that all of soft science is dodgy doesn't really distinguish us from them, does it? We do a lot of soft science!

Re: The Irreproducibility Crisis of Modern Science

#210

Post by lisper »

3JPLW wrote: post #200
lisper wrote: post #135

Earlier quoted context omitted.

> > at least one result in 20 will be due to chance > That's wildly overly pessimistic Why? You yourself said: > The null hypothesis is correct but they got "unlucky" data (5% chance or p% chance) How is that different from what I said? 5% is just another way of writing 1 in 20.

It's different because you're ignoring my second bullet point. There's a very common misconception that p-value is the percent chance the data confirm that the effect is real. That's not at all how p-values work. They are showing the _likelihood_ of generating the data were there no "effect." You make a "null hypothesis" — that is, you assume there is no effect — and you construct a distribution of the results you mi…

I think you're just misinterpreting what I mean by "failure" or "wrong result". I mean a positive result that is due to chance rather than to the null hypothesis actually being false. On that view, statistics alone guarantee a "failure" rate of at least 1 in 20. That's what choosing a threshold of 0.05 means.

The on top of that baseline rate of false results you also have self-selection and occasional outright cheating that drives the failure rate higher than 1 in 20. 1 in 20 is what you get when you use a 0.05 threshold and everything else in the system works flawlessly.

Post reply on HN