Is there a similar study done on the physical sciences? I’m getting a bit of holier-than-thou feeling from this article. Edit: from all this talk of reproducibility, I wonder what percentage of cutting edge ML research is reproducible (either from lack of public training sets / not enough compute)
What's wrong with social science and how to fix it
81–90 of 105 posts
Re: What's wrong with social science and how to fix it
#82There were some stunning claims being made on Twitter last month based on a recently published study. Instantly skeptical, I dug into the methodology section and found this gem: "It should be noted that the results cannot be estimated using a physician fixed effect due to a numeric overflow problem in Stata 15 which cannot be overcome without changing the assumptions of the logit model." ... The sad part was they did…
(Ignore my previous reply I found it myself). To be fair to the authors, it is not their primary specification, that was a linear probability model. The logit model is just a robustness check to make sure the linearity assumption isn't driving the results.
Re: What's wrong with social science and how to fix it
#83Does anyone have links to the Replication Prediction Market results mentioned in the article? That sounds super interesting. As an amusing nudge, I bet you could do some ML to predict replicability of a paper (per author's suggestion that it's laughably easy to predict) and release that as a tool for authors to do some introspection on their experimental design (assuming they're not maliciously publishing junk).
Re: What's wrong with social science and how to fix it
#84I am glad this topic is getting attention. There is significant bias in academia in social science even outside flaws in statistical techniques. The field has been weaponized to build foundational support for political stances and blind institutional trust granted to academia is enabling it. This author mentions the implicit association test (IAT) as an example of a social science farce that is well known to be a far…
Most analyses of the problems in science are really analyses of the problems in academia. There's no iron law that states academia has to be funded to the level it is today, and for most of history it wasn't. And let's recall, that these meta-studies are all about science, which is one of the better parts of academia. Once you get outside that into English Lit, gender studies, etc, the whole idea of replicating a paper ceases to be meaningful because the papers often aren't making logically coherent claims to begin with.
A lot of people look down on corporate funded science, but it has the huge advantage that discoveries are intended to be used for something. If the discovery is entirely false the resulting product won't work, so there is a built-in incentive to ensure research is telling you true things. The downside is there's also no incentive to share the findings, but that's what patents are for.
Of course a lot of social psych and other fields wouldn't get corporate funding. But that's OK. That's because they aren't really useful except for selling self-help books, which is unlikely to be a big enough market to fund the current level of correlational studies. That would hardly be a loss, though.
Re: What's wrong with social science and how to fix it
#85I guess I fall under the field of "Progress Studies" though I think I'm much less concerned with the replication crisis than most. Most new social science research is wrong. But the research that survives over time will have a higher likelihood of being true. This is because a) it is more likely to have been replicated, b) its more likely to have been incorporated into prevailing theory, or even better, have survived…
Not if this article is to be believed! He claims that studies that could not be replicated are about as likely to be cited as studies which are. That implies the problem may instead get worse and worse, the structure more and more shaky as time goes on.
Re: What's wrong with social science and how to fix it
#86There were some stunning claims being made on Twitter last month based on a recently published study. Instantly skeptical, I dug into the methodology section and found this gem: "It should be noted that the results cannot be estimated using a physician fixed effect due to a numeric overflow problem in Stata 15 which cannot be overcome without changing the assumptions of the logit model." ... The sad part was they did…
(Ignore my previous reply I found it myself). To be fair to the authors, it is not their primary specification, that was a linear probability model. The logit model is just a robustness check to make sure the linearity assumption isn't driving the results.
The Stata thing was just one of many, many red flags.
Re: What's wrong with social science and how to fix it
#87There were some stunning claims being made on Twitter last month based on a recently published study. Instantly skeptical, I dug into the methodology section and found this gem: "It should be noted that the results cannot be estimated using a physician fixed effect due to a numeric overflow problem in Stata 15 which cannot be overcome without changing the assumptions of the logit model." ... The sad part was they did…
I mean they couldn’t just do it in Matlab? or Python? So incredibly lazy.
Re: What's wrong with social science and how to fix it
#88Re: What's wrong with social science and how to fix it
#89Earlier quoted context omitted.
Might be that I am older than most of you, but when I was in high school the term was "social studies" then later it evolved to "social science" so the university crowd could legitimize it to get more funding. there obviously is no science in it.
"Social studies" refers to a variety of different social science and humanities subjects taught in public education. The term comes from US lawmakers around 1900. "Social sciences" are generally considered the "science of society," and the term comes from philosophers in the mid-1800's.
Some of the philosophers named it "science" of society in order to piggyback off the reputation of real science.
Because real science has such a great reputation, everyone from creationists ( creation science ) to social "scientists" ( social science ) have tried to associated themselves with science to gain credibility.
Social science isn't science. Neither is creationism.
Real science ( as most people understand it ) deal with natural laws and the natural world.
Re: What's wrong with social science and how to fix it
#90Earlier quoted context omitted.
"Social studies" refers to a variety of different social science and humanities subjects taught in public education. The term comes from US lawmakers around 1900. "Social sciences" are generally considered the "science of society," and the term comes from philosophers in the mid-1800's.
> "Social sciences" are generally considered the "science of society," and the term comes from philosophers in the mid-1800's. Some of the philosophers named it "science" of society in order to piggyback off the reputation of real science. Because real science has such a great reputation, everyone from creationists ( creation science ) to social "scientists" ( social science ) have tried to associated themselves with…
They very much are a part of the natural world, and presumably have some kind of "natural laws" guiding them. Creationism is not, as it is about things outside our observable world