Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's wrong with social science and how to fix it

fantasticanachronism.com

Post reply on HN
Print view View original

91–100 of 105 posts

Re: What's wrong with social science and how to fix it

#91

Post by garden_hermit »

karaterobot wrote: post #85
garden_hermit wrote: post #70

I guess I fall under the field of "Progress Studies" though I think I'm much less concerned with the replication crisis than most. Most new social science research is wrong. But the research that survives over time will have a higher likelihood of being true. This is because a) it is more likely to have been replicated, b) its more likely to have been incorporated into prevailing theory, or even better, have survived…

> This is because a) it is more likely to have been replicated, b) its more likely to have been incorporated into prevailing theory, or even better, have survived a shift in theory Not if this article is to be believed! He claims that studies that could not be replicated are about as likely to be cited as studies which are. That implies the problem may instead get worse and worse, the structure more and more shaky as…

Citation is not an endorsement—plenty of things are disagreed with in order to disagree with something, reference history in a field, or contextualize a result against past findings.

Here, the author seems to only look at recent papers, and so we don't really get to see how the citation patterns have evolved over 10, 20, or 30 years. But even then, established ideas tend to not be cited at all— the concept of "knowledge spillovers", for example, is common in Economics and other fields, yet the original reference is rarely used. Other times, more established claims will be encoded in a book or some work of theory—and people will cite the theory rather than the paper that made the original claim.

Re: What's wrong with social science and how to fix it

#92

Post by SubiculumCode »

jonnycomputer wrote: post #75
SubiculumCode wrote: post #47

Earlier quoted context omitted.

Indeed, and even if the replication isn't significant, it doesn't mean that the replication and the original study significantly differ from each other. Overall, the condescending trash talking in this article led me to flag it.

Could you speak a little more on that point?

bruh

Re: What's wrong with social science and how to fix it

#93

Post by stewbrew »

xrisk wrote: post #45
stewbrew wrote: post #42

The "social sciences" include a lot. Wrt Sociology, I'd say one problem is the overemphasis on quantitative methods - they try to be as serious as the big boys. The best sociological research I've read was qualitative though. Questionable replicability is of course built-in in this type of research but the research dealt with relevant questions. Most quantitative sociology seems rather irrelevant to me. Another probl…

I’m really interested in what you feel about the potential applications of CS/ML to sociology. Or if you might have any resources that talk about that. I ask because I’m enrolled in a research program in “computational humanities”. My initial feeling towards the program is that it’s kind of a sham. Computational Humanities seems to be as computational as an accountant using Excel for their work. Not that I particular…

There is a Springer "Journal of Computational Social Science". That could be a source of inspiration. CS/ML in Social Sciences gets interesting where a great amount of routine data is generated -- i.e. areas close to public administration.

Why did you subscribe in the program when you're not interested in the computational aspect? Or are you more interested in some kind of grand social theory/philosophy of computation? If you read German, Armin Nassehi "recently" published "Muster. Theorie der digitalen Gesellschaft" (Pattern. Therory of a digital society). He is not the first but I find his stance interesting - based on several interviews, I haven't read the book though. Many sociologists deal with the Internet & AI but I find those works less inspiring because they usually lack an adequate technical understanding. To me it often feels like bushmen theorizing about empty coca cola bottles (you probably don't know the movie?).

Re: What's wrong with social science and how to fix it

#94

Post by pas »

textman wrote: post #69
goatinaboat wrote: post #57

Earlier quoted context omitted.

You basically need to pay homage to the "ground breakers" who claimed that turf first, even if they did it via fraud. Even referring to it as “science” is fraudulent. Testable theories and repeatable outcomes, anyone? Time this whole field was defunded.

Might be that I am older than most of you, but when I was in high school the term was "social studies" then later it evolved to "social science" so the university crowd could legitimize it to get more funding. there obviously is no science in it.

That's just ignorance of how science works. There are no experiments in astronomy either ... or are there?

Natural experiments. Just as much as they happen in economics and sociology. And even though stars and black holes might be simpler than humans, it doesn't mean the soft sciences are not science.

That said exactly because of the additional complexity the bar should be higher for what's acceptable as research, viable methods and appreciable results.

Re: What's wrong with social science and how to fix it

#95

Post by pas »

thu2111 wrote: post #83
mNovak wrote: post #24

Does anyone have links to the Replication Prediction Market results mentioned in the article? That sounds super interesting. As an amusing nudge, I bet you could do some ML to predict replicability of a paper (per author's suggestion that it's laughably easy to predict) and release that as a tool for authors to do some introspection on their experimental design (assuming they're not maliciously publishing junk).

IIRC from prior discussions of this, a lot of the accuracy of the markets comes from people just applying common sense - like, if a really surprising claim that people should really have noticed before now comes with a huge effect size, it's probably false. ML can't judge that because it doesn't have the ability to do basic sanity checks on claims like that. It takes a sceptical human with life experience to do that.

Huh? That sounds exactly the thing that a ML system would learn quickly from the data. You probably don't even need shiny deep learning (though it helps).

Just like with the Netflix Prize stuff, where the conclusion was very similar, ie. just dump in as much data as you can, crank up the ML machinery, and it'll discover the features (better than you can engineer them) and learn what to use for recommendation ranking. And that's basically what we see with GPT-3 too. If you have some useful labels in the data it'll learn them even without supervision, because it has so many parameters, it basically sticks.

Get some papers run it through a supervised training phase where you give it a set with every paper scored based on how retracted/bad/unreplicating it is and you'll get a great predictor. Then run it to find papers that stick out, and then have a human look at them, and try to replicate some of them to fine-tune the predictor. Plus continue to feed it with new replication results.

That said, using an ML system as the gatekeeper as OP suggested is a bad idea, as it'll quickly result in the loss of proxy variables' predictive power.

Though ultimately a GPT-like system has the capacity to encode "common sense".

Re: What's wrong with social science and how to fix it

#96

Post by disown »

boomboomsubban wrote: post #90
disown wrote: post #89

Earlier quoted context omitted.

> "Social sciences" are generally considered the "science of society," and the term comes from philosophers in the mid-1800's. Some of the philosophers named it "science" of society in order to piggyback off the reputation of real science. Because real science has such a great reputation, everyone from creationists ( creation science ) to social "scientists" ( social science ) have tried to associated themselves with…

Society is an observable phenomenon that we are able to run experiments on and acquire empirical evidence about. Those experiments have so far been much more difficult to run, rely on poor measuring tools, and the limited data we have acquired is given too much weight. That's still science though. They very much are a part of the natural world, and presumably have some kind of "natural laws" guiding them. Creationism…

> Society is an observable phenomenon that we are able to run experiments on and acquire empirical evidence about.

Yes we can. But not in the scientific way. Society changes. You cannot replicate your test on a society because of that. Natural laws do not. So we can reliably hypothesize and test it.

> That's still science though.

It's not a science. By definition.

> They very much are a part of the natural world, and presumably have some kind of "natural laws" guiding them.

Yes. They have natural laws guiding them. It's called physic, chemistry, biology, etc. Social "science" doesn't delve into "natural laws". Society is run, not on natural laws, but human laws.

> Creationism is not, as it is about things outside our observable world.

No it is not. It is about our observable world but uses the bible ( another thing in the natural world ).

Laws, history, political science, economics, etc are are real sciences. They are political/propaganda/etc tools to shape society.

Using your logic, just because literature exists in the natural world, it must mean literature is a science. Obviously it is not.

Just because you can observe society doesn't make it a science, no more than the fact that you can observe a play or a concert makes them a science.

In order for your claim to hold, society has to be constant. But they are not. We must have multiple copies of the same society to test on. We do not. And societies are guided not by natural law, but human law. Which leads to an absurdity because different human societies have different laws. If we follow your logic, then means that there are different natural laws depending on where you live and what country you are from.

For example, that there are laws against interracial or interreligious marriages in some societies isn't a natural law, it's a human law. There is no science in social science. No more than there is science in religion or astrology.

Re: What's wrong with social science and how to fix it

#97

Post by mkatx »

disown wrote: post #96
boomboomsubban wrote: post #90

Earlier quoted context omitted.

Society is an observable phenomenon that we are able to run experiments on and acquire empirical evidence about. Those experiments have so far been much more difficult to run, rely on poor measuring tools, and the limited data we have acquired is given too much weight. That's still science though. They very much are a part of the natural world, and presumably have some kind of "natural laws" guiding them. Creationism…

> Society is an observable phenomenon that we are able to run experiments on and acquire empirical evidence about. Yes we can. But not in the scientific way. Society changes. You cannot replicate your test on a society because of that. Natural laws do not. So we can reliably hypothesize and test it. > That's still science though. It's not a science. By definition. > They very much are a part of the natural world, and…

Math is the foundation of science, right? Things in science are provable because of the rigorous mathematical proofs applied to data, correlating with reproducible results, is my perception.

There are many models of societies currently and previously in existence that can be measured, compared, and correlated with predictable cause and effects.

Society is not constant, it's probabilistic, like many topics in physics or other hard sciences.

Society has natural laws that can be mathematically derived and applied. Why can't science be applied to non natural processes? The point of science is to build a model that can be used to build other models, or ultimately be applied to game the subject.

Are you saying we learn physics just to learn it, not to manipulate it to somehow benefit ourselves? Don't we use research in physics as a tool to shape our world in exactly the same way we use research in social science to shape society?

Your definition of science seems to only exclude some folks from the term that are doing the exact same work in a different subject. The word science doesn't preclude a subject. I don't think your contrast is sound.

Re: What's wrong with social science and how to fix it

#98

Post by dleslie »

thu2111 wrote: post #84
throwawaysea wrote: post #30

I am glad this topic is getting attention. There is significant bias in academia in social science even outside flaws in statistical techniques. The field has been weaponized to build foundational support for political stances and blind institutional trust granted to academia is enabling it. This author mentions the implicit association test (IAT) as an example of a social science farce that is well known to be a far…

It could just be abolished. If the NSF was scrapped, what would the scientific world look like? Probably like the world before WW2, when most science was done by hobbyists or "inventors". Einstein did his seminal work whilst a patent clerk, etc etc. Most analyses of the problems in science are really analyses of the problems in academia. There's no iron law that states academia has to be funded to the level it is tod…

> If the NSF was scrapped, what would the scientific world look like? Probably like the world before WW2, when most science was done by hobbyists or "inventors".

There were scientists who received financial backing from wealthy individuals in a manner not so different than VCs operate today; Tesla among them.

Regardless, I tend to agree that science that exists for the sake of publishing, because publishing is a requirement of receiving grants, has diluted the respectability of science.

Re: What's wrong with social science and how to fix it

#99

Post by thu2111 »

pas wrote: post #95
thu2111 wrote: post #83

Earlier quoted context omitted.

IIRC from prior discussions of this, a lot of the accuracy of the markets comes from people just applying common sense - like, if a really surprising claim that people should really have noticed before now comes with a huge effect size, it's probably false. ML can't judge that because it doesn't have the ability to do basic sanity checks on claims like that. It takes a sceptical human with life experience to do that.

Huh? That sounds exactly the thing that a ML system would learn quickly from the data. You probably don't even need shiny deep learning (though it helps). Just like with the Netflix Prize stuff, where the conclusion was very similar, ie. just dump in as much data as you can, crank up the ML machinery, and it'll discover the features (better than you can engineer them) and learn what to use for recommendation ranking.…

Even GPT-3 doesn't encode common sense, which is why it can't do a lot of basic physical reasoning. It's "just" word prediction, albeit very impressive word prediction.

If you look at what GPT produces closely, a lot of it is simply bullshit. It sounds plausible but is wrong. That's exactly the wrong type of AI to detect plausible-but-wrong-bullshit papers, which are the most common type.

Re: What's wrong with social science and how to fix it

#100

Post by disown »

mkatx wrote: post #97
disown wrote: post #96

Earlier quoted context omitted.

> Society is an observable phenomenon that we are able to run experiments on and acquire empirical evidence about. Yes we can. But not in the scientific way. Society changes. You cannot replicate your test on a society because of that. Natural laws do not. So we can reliably hypothesize and test it. > That's still science though. It's not a science. By definition. > They very much are a part of the natural world, and…

Math is the foundation of science, right? Things in science are provable because of the rigorous mathematical proofs applied to data, correlating with reproducible results, is my perception. There are many models of societies currently and previously in existence that can be measured, compared, and correlated with predictable cause and effects. Society is not constant, it's probabilistic, like many topics in physics…

> Math is the foundation of science, right?

No. The scientific method is the foundation of science. Math is a wonderful tool that is used in science and other fields like business, architecture, music, etc.

> Things in science are provable because of the rigorous mathematical proofs applied to data, correlating with reproducible results, is my perception.

Nothing in science is "provable". Science is not math or logic. Go look up what a scientific theory is.

> The point of science is to build a model that can be used to build other models, or ultimately be applied to game the subject.

That is not the point of science. It is a benefit of science but not the point of science.

> Your definition of science seems to only exclude some folks from the term that are doing the exact same work in a different subject.

Not my definition. I didn't make it up. Just because you haven't a clue what science really is doesn't mean it's "my" definition. How about you take a few seconds and relearn what was taught in high school.

You have a bizarre notion that science exists to benefit ourselves. Religion, art, literature, music, business, etc also benefit us. Doesn't make it science.

Society is a human creation with human laws. That is no more "natural law" than the man-made laws governing chess or monopoly are "natural law".

Now that doesn't mean social "science" is worthy of study. Of course, humanities are worthy of study. But it isn't a science. I think politics is just as valid a field of study as physics or art or literature. But it doesn't make political science a science.

But your real worry is that removing science from social science takes away much of the credibility and respect that real scientists in physics, biology, chemistry, etc have earned right? In my experience, it's usually the social "scientists" who are the most vocal about latching onto "science" because they so desperate want cling onto that unearned authority and credibility.

But then again, it's also why creationists ( creation science ) also want to latch on to science. Pretty much frauds who want to manipulate society want something to give them credibility.

Post reply on HN