This is a tangent. How much/long does a study like this cost/take? Is it possible/useful to scale this? From reading into some of the replicability discussion, I've gotten curious about social science generally. What's an expirement like this trying to prove/demonstrate. The journalistic narrative likes the easy "X debunked." I imagine researchers have a more nuanced perspective. I guess what im asking is what does t…
That's not really possible. So instead the experiments that are made are toy experiments, but when you're not really testing what you're trying to measure, it becomes impossible to prove anything, and possible to show anything. For instance could you design a toy experiment that might indicate video games cause violence? Of course. Could you design a toy experiment that might indicate video games don't cause violence? Of course. The experiments are meaningless.
As an example of the problem, 'emotional intelligence' in work is all the rage right now. Yet the keystone study that sparked it is really just quite bad. The author had people split off into groups and perform a variety of tasks such as, literally, planning a shopping list. Using some method of determining who made the best shopping list, the author then determined that the groups who had the highest average IQ did not perform the best, whereas their 'emotional intelligence' as measured by an, again literally, "reading the mind in the eyes test" mapped better to performance - so therefore, it's not merit alone that judges performance but some emotional intelligence, at least as measured by "reading the mind in the eyes" that does. That's just broken logic. At the bare minimum IQ != specific task merit. The most logical way to perform that experiment would be to have created teams of those that performed individually best on any given task to work together against those that did not score so well on merit, but did well on the "reading the mind in the eyes test". Of course she did not do this, the bare minimum to even begin approaching this question, since the result would not be what she wanted. And negative results don't get published. Yet now there have been likely hundreds of articles and spin off studies taking that study's unjustified conclusion as a granted.
So no, there is absolutely no big picture progress in the social sciences.