"In order to be a true scientist, you should be familiar with philosophy, first"
The Irreproducibility Crisis of Modern Science
141–150 of 265 posts
Re: The Irreproducibility Crisis of Modern Science
#142Earlier quoted context omitted.
>I think when statistics is involved, the results are inherently untrustworthy. Ummm...are you kidding? Statistically vetted results are inherently UNCERTAIN, but how could they possibly be inherently untrustworthy? Even if a mechanistic effect is observed, its relationship to a particular cause or influence is only established statistically. In fact, the very observation is often performed u Dee the umbrella of stat…
I cannot speak for the person you are responding to, but in my [agreement] of his critique of statistics, I am implicitly speaking of social statistics. I think there is a vast difference in e.g. a statistical modeling of the behavior of electrons and e.g. the statistical modeling of some sort of human behavior.
...but that just isn't true. Statistical analysis is (among other things) a way to quantify our uncertainty. If done approriately, the statistics simply communicate the role probability played in moving from the experimental premises to the results.
The level of uncertainty in most (perhaps all) experiments involving particles in a vacuum is far lower than experiments involving human behavior. Statistics doesn't create the uncertainty, it communicates the uncertainty.
I can use propositional logic to arrive at the conclusion that "Unicorns are real" yet I don't encounter many people throwing logic itself under the bus because of that. Statistics has been a whipping boy for a long time by people who don't know much about statistics or just fail to think clearly about what is being said.Re: The Irreproducibility Crisis of Modern Science
#143Recently I have been beginning to question university education. It seems that today's education system has mismatched priorities which may have an impact on the reproducibility crisis. You see, to come up with a fantastic new alloy, you don't have to be a genius. Or even very good. You have to be persistent, and clever enough to realize when you've found something interesting. You will need to understand your field…
[deleted]
Re: The Irreproducibility Crisis of Modern Science
#144I have met several "refugees" from particle physics, that left in part because of flimsy statistical methods. So let me ask: who is going to reproduce the experiment that found the Higgs boson?
Re: The Irreproducibility Crisis of Modern Science
#145Lets take a page from Marx. Science is many things, in particular a relationship between capital and labor. The scientific method is a wonderful idea, but it is subordinate to the economic forces that underlie scientific activity. Look at the conflicts and contradictions between those doing science (labor) and those deciding the science to be done (capital), and that is the ultimate source of these crises. The execut…
The reproducibility crisis is most severe in the social sciences. Hard sciences like physics are on much firmer ground, and conflating the two is clownish. Social science research is funded almost entirely by the government[1]. It's a nice story you've got there, but the reason there is a replication crisis in the softer sciences is not due to the evil capitalists of the marxist imagination. In fact, much of it is du…
Maybe none of this ideological bullshit is the real issue? Maybe the major problem is that hard vs soft sciences are subject to different experimental controls and confounding factors, not to mention standards of rigor. If you claim to discover a new particle your n will be in the billions, and a whole community will analyze your results with a fine comb. Meanwhile if you make a claim about implicit bias or the nutritional value of coffee it’s based on tiny samples with huge variability.
It’d be cool if your and the other poster’s ideological spat could try to not use science as the battleground, or at least eliminate more likely issues before going there. Social sciences are hardly even science from the perspective of a chemist or physicist, there’s no need to delve into petty ideology. Methodology is a much richer source of hurdles to overcome than old and sad debates.
Re: The Irreproducibility Crisis of Modern Science
#146Earlier quoted context omitted.
>The data was bad, the processes were bad, there was no replication. People are really quick to say this, but how many people have actually studied the 'racial science' literature of the 20s and 30s and figured out to what extent it meets the methodological standards of today? Yes, we know that they drew incorrect (and reprehensible) conclusions. It doesn't necessarily follow that their methods were any worse (episte…
Studies about people in general are of poor quality, due to difficult to define metrics. Even today, psychology and sociology studies aren't rigorous for the simple reason that it's hard to measure these things. Biological studies are hard too, it's hard to suss out the confounding variables and mechanisms of bodily functions. Just the premise of the "racial science" studies would render them invalid, how could they…
Which studies are you referring to? If you don't have anything specific in mind, then you can't really judge the quality or compare it to the quality of comparable research today.
Re: The Irreproducibility Crisis of Modern Science
#147Earlier quoted context omitted.
I think we're passed the point where studies can be reproduced effectively. I actually think the scientific model is falling apart. We've discovered the easy things, and those studies were easy to reproduce. We built a model of science around the low hanging fruit. How are you going to reproduce a study about a cancer drug? Synthesizing the drug itself is a huge endeavor, never mind finding a group of participants wi…
> I actually think the scientific model is falling apart. You must have a totally different definition of "the scientific model" than I do. I suspect (although I could be wrong) is that you have an oversimplified view of just what the scientific process is. The scientific model/process is not a linear list of discrete steps moving from "Ask a question" to "Communicate results"; it is not a discrete and static set of…
Ok, I read your link. I still think my comments stands. My argument would be that the "Community Analysis and Feedback" section is practically impossible to maintain.
Back in early science days, this used to be a valuable section because the pool of "scientists" was huge and they often crossed disciplines. Now you have all sorts of specialties such that the pool of colleagues who are able to understand and assess your work might be, like, 8 people.
There was a Japanese mathematician who recently put out a huge proof that claimed to solve a major math problem (I am forgetting all of the important details here, so forgive me). The news wasn't so much that he created the proof, but that they couldn't find anyone to verify it!
Re: The Irreproducibility Crisis of Modern Science
#148Lets take a page from Marx. Science is many things, in particular a relationship between capital and labor. The scientific method is a wonderful idea, but it is subordinate to the economic forces that underlie scientific activity. Look at the conflicts and contradictions between those doing science (labor) and those deciding the science to be done (capital), and that is the ultimate source of these crises. The execut…
> There's just a very big gap between how science should be done and what capital owners want from science It was "a team from Bayer Healthcare" who "tried to replicate the results of basic cancer studies," failed, "and kicked off a media storm questioning the legitimacy of cancer science—and science in general" [1]. The "capital owners" looking out for their own buck are performing more effectively, narrowly speakin…
Even if that's true, that only science that can have a profit motive slapped on it can be done effectively in the current system is a problem. If only such science were done, then we would have different crises of science to deal with because profit seeking is at odds with much science, in particular basic research.
It's also quite obviously the case that profit seeking research has incentives to be bad science in other ways.
Re: The Irreproducibility Crisis of Modern Science
#149What I'd like to know is if the "irreproducibility crisis" is really some combination of "sample size was too small" and "effect size was too small". When I went through a lot of these studies myself and saw the ones that don't reproduce, I saw this theme over and over. "P-hacking" is less of a concern to me when the effect is real and widespread. It's so bad now that for any article/study, I look at the sample size…
Huge sample studies with high stat significance can have very low effect sizes, though. If an effect is shown on a small sample, it means it is a really big effect, and will likely be reproduced in a bigger study.
Re: The Irreproducibility Crisis of Modern Science
#150Earlier quoted context omitted.
I think this is pretty self evident, but the issue is that let's say you have some system where economic forces are removed. Essentially a researcher basic income in one scenario. This would suddenly massively incentivize people towards this direction since it's basically a career path that guarantees a stable livelihood, which is something that's extremely rare today. Well you need to ensure there's nobody just comp…
Small correction. A basic income is guaranteed no matter what; there’s no means testing or anything like that. You can’t game the system because there’s nothing to game: you get the money regardless. You seem to be describing the system we have now, where you’re paid for results.
A universal basic income is guaranteed no matter what; but it's certainly possible to have basic income schemes that are not universal but limited to some (small) subpopulation, and the criteria for being in that subpopulation can be gamed.