Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Irreproducibility Crisis of Modern Science

nas.org

Post reply on HN
Print view View original

231–240 of 265 posts

Re: The Irreproducibility Crisis of Modern Science

#231

Post by [deleted] »

foldr wrote: post #225
lotsofpulp wrote: post #202

Earlier quoted context omitted.

It's not going wrong, unless it's making conclusive statements that are beyond the strengths of the study. That's why I say grain of salt. I understand what you're saying, but my point is that it's not science or the scientific method that results in undesirable outcomes, such as the Nazi example, but just a lack of any other good option. Governments (and people) should evaluate the merits of the information they rec…

>Properly conducted and repeated experiments with noted mechanisms of actions (e.g. vaccines) This is a good example of why it's not so simple as you're making out. Earlier, you put biology on your list of unreliable branches of science, and you claimed that you could know that the racial science of the 30s wasn't "real science" simply because it consisted of "studies about people". But vaccine science falls into bot…

[deleted]

Re: The Irreproducibility Crisis of Modern Science

#232

Post by dang »

spikels wrote: post #224
dang wrote: post #204

Earlier quoted context omitted.

Please don't take HN threads on generic ideological tangents, which never lead anywhere new. I know the GP pointed there, but that's no reason to follow the pointer. https://news.ycombinator.com/newsguidelines.html

Fair enough but I was reacting to the clearly idealogical. To suggest Marx has something to add in this issue is both provocative and ridiculous. To entertain drivel this degrades HN.

As I tried to explain, it's a matter of degree. If a comment contains both bait and substance, it falls to the rest of us not to take the bait, since doing that guarantees a useless tangent or worse. The acceptable options are either (1) react to the substance, or (2) don't react.

Getting triggered happens to everyone but is not an act of intellectual curiosity. https://news.ycombinator.com/newsguidelines.html

Re: The Irreproducibility Crisis of Modern Science

#233

Post by lisper »

3JPLW wrote: post #230
lisper wrote: post #229

Earlier quoted context omitted.

> it can’t evaluate which of two competing cases is more likely Of course it can. In fact, it can tell you exactly how much more likely one case is versus the other. P=0.05 means that there is a 5% chance that the null is true and hence a 95% chance the the null is false. > says something about the likelihood of the _hypothesis_ (like you're trying to do) No. I am simply saying that there is a lower bound on the rate…

Were I to run 100 independently designed experiments that all tested real effects, my choice of p-value does not determine the number that erroneously find no result. If the effect is small and I didn't gather enough data, a p-value of 0.05 could result in only a handful of experiments accurately reflecting reality. Let's say 30 make the cut. Were I to run another 100 independently designed experiments that all teste…

> Were I to run 100 independently designed experiments that all tested real effects, my choice of p-value does not determine the number that erroneously find no result.

That's actually not true. The problem is that you cannot define what is a "real effect" without begging the question. Let me illustrate with an example: Suppose I do what appears to be a legitimate experiment to test a well-accepted law of nature. Unbeknownst to me, my instrument is faulty (or has been tampered with) and is in fact returning essentially random results. If I choose pcorrectly finding the results of a different experiment than the one I think I'm running.

And, of course, an instrument that returns random numbers is an extreme example. See:

https://arstechnica.com/science/2018/04/new-measurements-set...

for a current real-world example.

Now, you do have a valid point in that I made a tacit assumption when I claimed that statistics put a lower bound on the error rate. That assumption is that most experiments that are done test hypotheses that ultimately turn out to be false. I believe this assumption is actually true. This is one of the reasons making scientific discoveries is hard. But I didn't actually state this assumption, so I'll cop to that.

> At this point you're arguing against the sources I cited.

That's not true either. The first source says that the error rate is much higher than the p-value, which agrees with me. (I'm actually a little skeptical that the error rate is as high as your source says -- "At least 23% (and typically close to 50%" for p=0.05, but that's certainly not refuting my position!) I don't have time to go through your second citation right now.

Re: The Irreproducibility Crisis of Modern Science

#234

Post by buvanshak »

haZard_OS wrote: post #142
TangoTrotFox wrote: post #116

Earlier quoted context omitted.

I cannot speak for the person you are responding to, but in my [agreement] of his critique of statistics, I am implicitly speaking of social statistics. I think there is a vast difference in e.g. a statistical modeling of the behavior of electrons and e.g. the statistical modeling of some sort of human behavior.

...but that just isn't true. Statistical analysis is (among other things) a way to quantify our uncertainty. If done approriately, the statistics simply communicate the role probability played in moving from the experimental premises to the results. The level of uncertainty in most (perhaps all) experiments involving particles in a vacuum is far lower than experiments involving human behavior. Statistics doesn't crea…

> If done approriately..

This is the whole point. It is the difficulty in doing so, or the difficulty in deciding if this was done appropriately or not is the only thing that it untrustworthy..

>People lie while speaking English every day yet I don't encounter many people claiming that English is inherently untrustworthy.

No one expect a statement to be true simply because it was uttered in English.

Re: The Irreproducibility Crisis of Modern Science

#235

Post by buvanshak »

lisper wrote: post #108
buvanshak wrote: post #93

Earlier quoted context omitted.

>For example, celestial events are almost never reproducible. Our understanding of celestial mechanics nonetheless rests on solid science. I am not sure this is correct. The rules that govern celestial bodies is same as the ones that govern objects on earth. So why are they not reproducible? It does not require to measure forces between celestial bodies to measure the value of G. Measuring the forces between two mass…

Yes, of course the rules are the same. Figuring this out was the event that launched the entire modern scientific endeavor. But nonetheless, the experimental data that went into this discovery was largely non-reproducible. The stars and planets go where they go and only very rarely does a given configuration repeat itself. In fact, no configuration ever really repeats itself in every detail. So, for example, we can p…

> But nonetheless, the experimental data that went into this discovery was largely non-reproducible.

I am finding a hard time to grasp what you are telling here. If you deduce a set of laws A from an event X, you don't need X to repeat to check the validity of A. You can check it against another event Y, where you predict its characteristics using A. If the laws are valid, the prediction ll match the actual observation..

Re: The Irreproducibility Crisis of Modern Science

#236

Post by gehwartzen »

geofft wrote: post #113
TangoTrotFox wrote: post #99

Earlier quoted context omitted.

I think this is pretty self evident, but the issue is that let's say you have some system where economic forces are removed. Essentially a researcher basic income in one scenario. This would suddenly massively incentivize people towards this direction since it's basically a career path that guarantees a stable livelihood, which is something that's extremely rare today. Well you need to ensure there's nobody just comp…

I don't see why you would want to solve this with a researcher basic income (and overhead/bureaucracy of figuring out who is a researcher) and not the simpler solution of an actual universal basic income.

I don't think a $10-20k/yr UBI is anywhere close to the actual cost of allowing individuals to conduct meaningful scientific studies.

Re: The Irreproducibility Crisis of Modern Science

#237

Post by lisper »

buvanshak wrote: post #235
lisper wrote: post #108

Earlier quoted context omitted.

Yes, of course the rules are the same. Figuring this out was the event that launched the entire modern scientific endeavor. But nonetheless, the experimental data that went into this discovery was largely non-reproducible. The stars and planets go where they go and only very rarely does a given configuration repeat itself. In fact, no configuration ever really repeats itself in every detail. So, for example, we can p…

> But nonetheless, the experimental data that went into this discovery was largely non-reproducible. I am finding a hard time to grasp what you are telling here. If you deduce a set of laws A from an event X, you don't need X to repeat to check the validity of A. You can check it against another event Y, where you predict its characteristics using A. If the laws are valid, the prediction ll match the actual observati…

Yes, all that is true. What does that have to do with reproducibility? You're talking about doing a lot of different experiments to test a single theory. Reproducibility is about doing the same experiment over again to see if you get the same results as the first time you did it, i.e. reproduce the results.

Re: The Irreproducibility Crisis of Modern Science

#238

Post by buvanshak »

lisper wrote: post #237
buvanshak wrote: post #235

Earlier quoted context omitted.

> But nonetheless, the experimental data that went into this discovery was largely non-reproducible. I am finding a hard time to grasp what you are telling here. If you deduce a set of laws A from an event X, you don't need X to repeat to check the validity of A. You can check it against another event Y, where you predict its characteristics using A. If the laws are valid, the prediction ll match the actual observati…

Yes, all that is true. What does that have to do with reproducibility? You're talking about doing a lot of different experiments to test a single theory. Reproducibility is about doing the same experiment over again to see if you get the same results as the first time you did it, i.e. reproduce the results.

> What does that have to do with reproducibility?

That is exactly what I wanted to ask you originally when you cited the celestial mechanics example. It is not relevant in cases we are deriving a theory underlying the behavior, rather than axiomizing the very specific behavior itself...

Re: The Irreproducibility Crisis of Modern Science

#239

Post by AstralStorm »

geofft wrote: post #161
JumpCrisscross wrote: post #136

Earlier quoted context omitted.

> There's just a very big gap between how science should be done and what capital owners want from science It was "a team from Bayer Healthcare" who "tried to replicate the results of basic cancer studies," failed, "and kicked off a media storm questioning the legitimacy of cancer science—and science in general" [1]. The "capital owners" looking out for their own buck are performing more effectively, narrowly speakin…

It certainly seems true that the owners of capital are suffering from the reproducibility crisis as much as anyone is. What 'danharaj is arguing, I think, is that the owners of capital are not well-placed to solve it. The very nature of capitalism is that people who produce monetizable results should be rewarded for that work, and that people who don't shouldn't. This is fundamentally incompatible with doing science…

Also capital owners would preferably not pay at all and have you work for free. They cannot approach this ideal only because the workers are not constrained spatially (like serfs) not forced (like slaves) to work for them. Sometimes they ate forced due to economic factors to do labour at marginal gain or even loss...

Re: The Irreproducibility Crisis of Modern Science

#240

Post by AstralStorm »

SubiculumCode wrote: post #176
PeterisP wrote: post #168

Earlier quoted context omitted.

The ensemble advantage is completely the other way around - we might choose to use a large ensemble of classifiers because they can get slightly better results than the best single strong classifier we can make; and we might choose not to use an ensemble because of computational cost reasons, especially because you train a model once but infer forever (and likely on more limited hardware) and inference for a hundred…

Always willing to learn more. I was going off of https://en.wikipedia.org/wiki/AdaBoost which mentions "Unlike neural networks and SVMs, the AdaBoost training process selects only those features known to improve the predictive power of the model, reducing dimensionality and potentially improving execution time as irrelevant features need not be computed." edit: When I read about ensemble theory, you receive support f…

Not quite. If you read the latter paragraph closely, you would note that an ensemble of stronger models performs better than a multitude of dumb models.

Then there is a problem that an ensemble of biased classifiers (mostly in the same ditection in this case - positivity bias) will magnify the bias.

This is a reason why metaanalyses have to get at the actual raw data to pool and analyse as well as correct for multiple biases. Even then the process is not perfect.

Post reply on HN