What's wrong with social science and how to fix it • Hacker News Board

Re: What's wrong with social science and how to fix it

#21

Post by jbay808 » Sat, Sep 12, 2020, 3:09 AM UTC

konjin wrote: post #8

not2b wrote: post #3

This guy overstates his case somewhat. Consider: "If the original study says an intervention raises math scores by .5 standard deviations and the replication finds that the effect is .2 standard deviations (though still significant), that is considered a success that vindicates the original study!" Why the exclamation point here? The replication study isn't magically more accurate than the original study. If the orig…

I'm sorry, but this is such a ridiculous counter argument I'm speechless. An explanation point as criticism? >How dare he wear tweed, his argument is invalid. >The replication study isn't magically more accurate than the original study. If the original paper finds an 0.5 standard deviation effect and the replication study finds an 0.2 standard deviation effect, that increases our confidence that a real effect was mea…

> An explanation point as criticism

As in "eats, shoots, and leaves", a little punctuation can totally change the meaning of a sentence. In this case, a period would have expressed agreement while an exclamation point expresses incredulity.

Re: What's wrong with social science and how to fix it

#22

Post by konjin » Sat, Sep 12, 2020, 3:10 AM UTC

jbay808 wrote: post #21

konjin wrote: post #8

Earlier quoted context omitted.

I'm sorry, but this is such a ridiculous counter argument I'm speechless. An explanation point as criticism? >How dare he wear tweed, his argument is invalid. >The replication study isn't magically more accurate than the original study. If the original paper finds an 0.5 standard deviation effect and the replication study finds an 0.2 standard deviation effect, that increases our confidence that a real effect was mea…

> An explanation point as criticism As in "eats, shoots, and leaves", a little punctuation can totally change the meaning of a sentence. In this case, a period would have expressed agreement while an exclamation point expresses incredulity.

!

Re: What's wrong with social science and how to fix it

#23

Post by konjin » Sat, Sep 12, 2020, 3:12 AM UTC

tomrod wrote: post #19

konjin wrote: post #9

Earlier quoted context omitted.

>Physics demands that we respect electricity and don't increase the infrared opacity of the atmosphere. Physics does no such thing. It tells us that increasing the heat retained in the atmosphere increases the planets surface temperature. It is a descriptive science. Not a prescriptive one. Wanting to have industrial civilization possible in the next century is why you don't increase the infrared opacity of the atmos…

This is why knowledge of causal inference is essential. Causality is not established via tweaking a correlation or regression analysis, and we social scientists should know that.

Casual inference is the bottom most rug of what gives hard sciences its power. It is that we understand the objects we are manipulating at a much deeper level so we don't sound like idiots.

Suppose that we take:

g = ma

A perfectly valid way to find experimental values for gravity at a location. But that doesn't mean that if we push an object really hard we increase the gravity in that location, or decrease it if we pull on the object. Just because symbol manipulation gives as an answer doesn't mean that the answer makes sense, you need to keep track of all the implicit state of the universe.

Re: What's wrong with social science and how to fix it

#24

Post by mNovak » Sat, Sep 12, 2020, 3:14 AM UTC

Does anyone have links to the Replication Prediction Market results mentioned in the article? That sounds super interesting.

As an amusing nudge, I bet you could do some ML to predict replicability of a paper (per author's suggestion that it's laughably easy to predict) and release that as a tool for authors to do some introspection on their experimental design (assuming they're not maliciously publishing junk).

Re: What's wrong with social science and how to fix it

#25

Post by stefan_ » Sat, Sep 12, 2020, 3:22 AM UTC

mhh__ wrote: post #13

I try not to look down on social science, for the most part data is data as long as you can reason about how it was collected and who by. The only thing that worries me a little (or a lot sometimes) is that there doesn't seem to be much "bone" for the meat to hang off of - that is, in physics, if your theory doesn't match experiment it's wrong whereas in social science you're never going to have a (mathematical) theo…

If only it was all data, statistics and hypotheses in social science. As it stands, that's a relatively modern idea in the field. Take a look at some of the stuff they teach in sociology 101:

https://en.m.wikipedia.org/wiki/Social_actions

It's all utter fiction. The long drawn out ramblings of ancient white men who formulated theories as their favorite pastime. None of it is experimentally derived because there were no experiments to begin with.

You can't found a field on a corpus of creative writing.

Re: What's wrong with social science and how to fix it

#26

Post by barry-cotter » Sat, Sep 12, 2020, 3:26 AM UTC

Kednicma wrote: post #4

> Even if all the statistical, p-hacking, publication bias, etc. issues were fixed, we'd still be left with a ton of ad-hoc hypotheses based, at best, on (WEIRD) folk intuitions. This is the quiet part which most social scientists, particularly psychologists, don't want to discuss or admit: WEIRD [0] selection bias massively distorts which effects are inherent to humans and which are socially learned. You'll hear peo…

The Big Five is a much less impressive accomplishment than you’d think for how much people talk about it.

https://carcinisation.com/2020/07/04/the-ongoing-accomplishm...

> The interesting thing about the Five Factor Model is what it gets away with, in terms of being considered a theory, even though it is not causal, and makes no predictions. What counts as a “replication” of the Five Factor Model, as in Soto (2019), is the following: a correlation is found between one or more factors of the Five Factor Model and some other construct, and that correlation is found again in another sample, regardless of the size of the correlation. In almost all cases, and in 100% of Soto (2019)’s measures, the construct compared to a Big Five factor is derived from an online survey instrument.

> What counts as a “consequential life outcome” is also fascinating. In most cases, the life outcome constructs are vague abstractions measured with survey instruments, much like the Big Five themselves. For instance, the life outcome “Inspiration” is measured with the Inspiration Scale, which asks the subject in four ways how often and how deeply inspired they are. Amazingly, this scale correlates a little bit with Extraversion and with Open-mindedness. Do these personality traits “predict” the life outcome of inspiration? Is “Inspiration” as instrumentalized here meaningfully different from the Big Five constructs, such that this correlation is meaningful?

Re: What's wrong with social science and how to fix it

#27

Post by throwawaysea » Sat, Sep 12, 2020, 3:29 AM UTC

jessriedel wrote: post #2

Summary tweet thread by the author: https://mobile.twitter.com/AlvaroDeMenard/status/13043994376...

Threadreader version: https://threadreaderapp.com/thread/1304399437641461760.html

He mentions changing the threshold for significance as a possible tweak but the issue is something more fundamental. Humans have flaws - like political biases or a tendency to favor one’s own hypotheses (confirmation bias). Humans also operate within systems that have incentives that can motivate them away from truth seeking (publication bias). All this exacerbated the fundamental problem that statistical techniques are easy to manipulate. Virtually all academic (university) studies, in their published format, simply lack the necessary information, controls, and processes a reader would need to easily detect flawed statistical claims. Instead a reader has to blindly trust - assuming that data was not selectively included/excluded or that the parameters of the experiment were rigorously (neutrally) chosen or whatever else. There is no incentive for the academic world to correct for this - there isn’t for example, a financial consequence for a decision based on bad statistics, as a private company might face.

Re: What's wrong with social science and how to fix it

#28

Post by throwawaysea » Sat, Sep 12, 2020, 3:34 AM UTC

mNovak wrote: post #24

Does anyone have links to the Replication Prediction Market results mentioned in the article? That sounds super interesting. As an amusing nudge, I bet you could do some ML to predict replicability of a paper (per author's suggestion that it's laughably easy to predict) and release that as a tool for authors to do some introspection on their experimental design (assuming they're not maliciously publishing junk).

Here’s the paper, published only recently: https://royalsocietypublishing.org/doi/10.1098/rsos.200566

> I bet you could do some ML to predict replicability of a paper (per author's suggestion that it's laughably easy to predict)

I am betting any such ML system could be gamed and addressing the issue would ultimately still need humans in the loop. For example, what if I am selective with my data, beyond the visibility of ML evaluating the final published paper? I don’t think this is “laughably easy” to predict. It may be easy to spot telltale signs today that predict replicability, but as soon as those markers are understood, I imagine authors will simply squeeze papers through the cracks in a different way.

Another issue is this bit from the author on Twitter:

> Just because it replicates doesn't mean it's good. A replication of a badly designed study is still badly designed. There are tons of papers doing correlational analyses yet drawing causal conclusions, and many of them will successfully replicate. Doesn't mean they're justified.

Re: What's wrong with social science and how to fix it

#29

Post by wrnr » Sat, Sep 12, 2020, 3:46 AM UTC

Lots of social science in crap, for sure, no arguing about it, dunno how to make it better then not to do it. Though, some are interesting if you have the patience for it, ex. Linguistics, Psychology and Economics, even things like critical theory are sort of useful, think of it like the abstract algebra of social science. Just people pulling apart concept to see if they can be put back together in another way to create something new. I now a lots of CS researchers and they do shit work and cite each others excrement, honestly CS is the sociology of STEM. Their I said it.

Re: What's wrong with social science and how to fix it

#30

Post by throwawaysea » Sat, Sep 12, 2020, 3:52 AM UTC

I am glad this topic is getting attention. There is significant bias in academia in social science even outside flaws in statistical techniques. The field has been weaponized to build foundational support for political stances and blind institutional trust granted to academia is enabling it. This author mentions the implicit association test (IAT) as an example of a social science farce that is well known to be a farce, and notes that most social science work is undertaken in good faith.

However the damage has been done and it doesn’t matter if MOST work is done in good faith if the bad work has big impact. As an example, IATs have been used to make claims about unconscious biases and form the academic basis of books like “White Fragility” by Robin DiAngelo. Quillette wrote about problems with White Fragility and IAT as early as 2018 (https://quillette.com/2018/08/24/the-problem-with-white-frag...), and others continue to write about it even recently in 2020 (https://newdiscourses.com/2020/06/flaws-white-fragility-theo...). However few people are exposed to these critical analyses, and the flaws in the scientific/statistical underpinnings have not mattered, and they have not stopped books like White Fragility from circulating by the millions.

We need a drastic rethink of academia, the incentives within it, and the controls that regulate it to stop the problem. Until then, it’s simply not worth taking fields like social science seriously.