Live data from Hacker News

What's wrong with social science and how to fix it

fantasticanachronism.com

81–90 of 105 posts

Re: What's wrong with social science and how to fix it

#81
post #38

Is there a similar study done on the physical sciences? I’m getting a bit of holier-than-thou feeling from this article. Edit: from all this talk of reproducibility, I wonder what percentage of cutting edge ML research is reproducible (either from lack of public training sets / not enough compute)

He mentions that epidemiology has actually more severe problems than economics. Having read some epi papers I understand why. Not sure if you'd count that as a physical or social science though: at least theoretically it's biologically based, but in reality the data it works with is mostly social and demographic.

Re: What's wrong with social science and how to fix it

#82
post #31

There were some stunning claims being made on Twitter last month based on a recently published study. Instantly skeptical, I dug into the methodology section and found this gem: "It should be noted that the results cannot be estimated using a physician fixed effect due to a numeric overflow problem in Stata 15 which cannot be overcome without changing the assumptions of the logit model." ... The sad part was they did…

(Ignore my previous reply I found it myself). To be fair to the authors, it is not their primary specification, that was a linear probability model. The logit model is just a robustness check to make sure the linearity assumption isn't driving the results.

When your robustness check fails because of numerical overflow...

Re: What's wrong with social science and how to fix it

#83
post #24

Does anyone have links to the Replication Prediction Market results mentioned in the article? That sounds super interesting. As an amusing nudge, I bet you could do some ML to predict replicability of a paper (per author's suggestion that it's laughably easy to predict) and release that as a tool for authors to do some introspection on their experimental design (assuming they're not maliciously publishing junk).

IIRC from prior discussions of this, a lot of the accuracy of the markets comes from people just applying common sense - like, if a really surprising claim that people should really have noticed before now comes with a huge effect size, it's probably false. ML can't judge that because it doesn't have the ability to do basic sanity checks on claims like that. It takes a sceptical human with life experience to do that.

Re: What's wrong with social science and how to fix it

#84

I am glad this topic is getting attention. There is significant bias in academia in social science even outside flaws in statistical techniques. The field has been weaponized to build foundational support for political stances and blind institutional trust granted to academia is enabling it. This author mentions the implicit association test (IAT) as an example of a social science farce that is well known to be a far…

It could just be abolished. If the NSF was scrapped, what would the scientific world look like? Probably like the world before WW2, when most science was done by hobbyists or "inventors". Einstein did his seminal work whilst a patent clerk, etc etc.

Most analyses of the problems in science are really analyses of the problems in academia. There's no iron law that states academia has to be funded to the level it is today, and for most of history it wasn't. And let's recall, that these meta-studies are all about science, which is one of the better parts of academia. Once you get outside that into English Lit, gender studies, etc, the whole idea of replicating a paper ceases to be meaningful because the papers often aren't making logically coherent claims to begin with.

A lot of people look down on corporate funded science, but it has the huge advantage that discoveries are intended to be used for something. If the discovery is entirely false the resulting product won't work, so there is a built-in incentive to ensure research is telling you true things. The downside is there's also no incentive to share the findings, but that's what patents are for.

Of course a lot of social psych and other fields wouldn't get corporate funding. But that's OK. That's because they aren't really useful except for selling self-help books, which is unlikely to be a big enough market to fund the current level of correlational studies. That would hardly be a loss, though.

Re: What's wrong with social science and how to fix it

#85

I guess I fall under the field of "Progress Studies" though I think I'm much less concerned with the replication crisis than most. Most new social science research is wrong. But the research that survives over time will have a higher likelihood of being true. This is because a) it is more likely to have been replicated, b) its more likely to have been incorporated into prevailing theory, or even better, have survived…

> This is because a) it is more likely to have been replicated, b) its more likely to have been incorporated into prevailing theory, or even better, have survived a shift in theory

Not if this article is to be believed! He claims that studies that could not be replicated are about as likely to be cited as studies which are. That implies the problem may instead get worse and worse, the structure more and more shaky as time goes on.

Re: What's wrong with social science and how to fix it

#86
post #31

There were some stunning claims being made on Twitter last month based on a recently published study. Instantly skeptical, I dug into the methodology section and found this gem: "It should be noted that the results cannot be estimated using a physician fixed effect due to a numeric overflow problem in Stata 15 which cannot be overcome without changing the assumptions of the logit model." ... The sad part was they did…

(Ignore my previous reply I found it myself). To be fair to the authors, it is not their primary specification, that was a linear probability model. The logit model is just a robustness check to make sure the linearity assumption isn't driving the results.

Yes, the primary specification was a linear probability model for the likelihood of a binary dependent variable conditioned on two binary input variables. As far as I could tell, the fit was max likelihood without regularization and the paper's bombshell conclusion was based on the regression coefficients' p-values.

The Stata thing was just one of many, many red flags.

Re: What's wrong with social science and how to fix it

#87
post #68
post #31

There were some stunning claims being made on Twitter last month based on a recently published study. Instantly skeptical, I dug into the methodology section and found this gem: "It should be noted that the results cannot be estimated using a physician fixed effect due to a numeric overflow problem in Stata 15 which cannot be overcome without changing the assumptions of the logit model." ... The sad part was they did…

I mean they couldn’t just do it in Matlab? or Python? So incredibly lazy.

Matlab is quote expensive.

Re: What's wrong with social science and how to fix it

#89
post #69

Earlier quoted context omitted.

Might be that I am older than most of you, but when I was in high school the term was "social studies" then later it evolved to "social science" so the university crowd could legitimize it to get more funding. there obviously is no science in it.

"Social studies" refers to a variety of different social science and humanities subjects taught in public education. The term comes from US lawmakers around 1900. "Social sciences" are generally considered the "science of society," and the term comes from philosophers in the mid-1800's.

> "Social sciences" are generally considered the "science of society," and the term comes from philosophers in the mid-1800's.

Some of the philosophers named it "science" of society in order to piggyback off the reputation of real science.

Because real science has such a great reputation, everyone from creationists ( creation science ) to social "scientists" ( social science ) have tried to associated themselves with science to gain credibility.

Social science isn't science. Neither is creationism.

Real science ( as most people understand it ) deal with natural laws and the natural world.

Re: What's wrong with social science and how to fix it

#90
post #89

Earlier quoted context omitted.

"Social studies" refers to a variety of different social science and humanities subjects taught in public education. The term comes from US lawmakers around 1900. "Social sciences" are generally considered the "science of society," and the term comes from philosophers in the mid-1800's.

> "Social sciences" are generally considered the "science of society," and the term comes from philosophers in the mid-1800's. Some of the philosophers named it "science" of society in order to piggyback off the reputation of real science. Because real science has such a great reputation, everyone from creationists ( creation science ) to social "scientists" ( social science ) have tried to associated themselves with…

Society is an observable phenomenon that we are able to run experiments on and acquire empirical evidence about. Those experiments have so far been much more difficult to run, rely on poor measuring tools, and the limited data we have acquired is given too much weight. That's still science though.

They very much are a part of the natural world, and presumably have some kind of "natural laws" guiding them. Creationism is not, as it is about things outside our observable world

Post reply on HN