Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's wrong with social science and how to fix it

fantasticanachronism.com

Post reply on HN
Print view View original

71–80 of 105 posts

Re: What's wrong with social science and how to fix it

#71

Post by garden_hermit »

bitxbit wrote: post #68
ims wrote: post #31

There were some stunning claims being made on Twitter last month based on a recently published study. Instantly skeptical, I dug into the methodology section and found this gem: "It should be noted that the results cannot be estimated using a physician fixed effect due to a numeric overflow problem in Stata 15 which cannot be overcome without changing the assumptions of the logit model." ... The sad part was they did…

I mean they couldn’t just do it in Matlab? or Python? So incredibly lazy.

Python, R, Mathab etc. are outside of many people's skillsets. I've tried to evangelize to many fellow researchers, but they simply don't have the time or the interest in novel programming languages when the tools they have (largely) work.

Re: What's wrong with social science and how to fix it

#72

Post by garden_hermit »

unishark wrote: post #40
xrisk wrote: post #38

Is there a similar study done on the physical sciences? I’m getting a bit of holier-than-thou feeling from this article. Edit: from all this talk of reproducibility, I wonder what percentage of cutting edge ML research is reproducible (either from lack of public training sets / not enough compute)

There are definitely studies criticizing ML publications similarly. As a kind of statistics (but often without the rigor), screwups make ML methods appear better than they really are. Hence the literature is packed with screwups. Other CS subfields that get a lot of criticism are "network science" and bioinformatics.

ML has the benefit though of a rapid turnaround time from Academia -> Industry. Things that work/replicate will be immediately used to make money. Things that don't work will be abandoned pretty quickly (at least outside of ML researchers).

Re: What's wrong with social science and how to fix it

#74

Post by jonnycomputer »

crazygringo wrote: post #65

I've tried to understand this (obviously quite angry/ranty) article and cannot actually figure out what data it has. It seems to not be based on actual replication results, but predicted replication results? But then the first chart isn't even predictions from the market, but just the author's predictions? The author clearly has a real hatred for practices in the social sciences. But I don't see any actual proof of t…

No, you are exactly right.

Re: What's wrong with social science and how to fix it

#75

Post by jonnycomputer »

SubiculumCode wrote: post #47
not2b wrote: post #3

This guy overstates his case somewhat. Consider: "If the original study says an intervention raises math scores by .5 standard deviations and the replication finds that the effect is .2 standard deviations (though still significant), that is considered a success that vindicates the original study!" Why the exclamation point here? The replication study isn't magically more accurate than the original study. If the orig…

Indeed, and even if the replication isn't significant, it doesn't mean that the replication and the original study significantly differ from each other. Overall, the condescending trash talking in this article led me to flag it.

Could you speak a little more on that point?

Re: What's wrong with social science and how to fix it

#76

Post by garden_hermit »

xrisk wrote: post #38

Is there a similar study done on the physical sciences? I’m getting a bit of holier-than-thou feeling from this article. Edit: from all this talk of reproducibility, I wonder what percentage of cutting edge ML research is reproducible (either from lack of public training sets / not enough compute)

There are tons of replication issues across the sciences, they are just most salient in the Social Sciences because the topic is just really hard to study well.

Clinical trials can often be flawed, even if the stats are fine, just in how they sample. For example, women are often excluded from trials due to hormonal changes, but how drugs impact women is really important! Participants are also typically drawn from specific locations, and so may not be representative of people with different diets, lifestyles, and environmental factors.

Physics has its own controversies, though not always directly related to replication. For example, Harry Collins recounts the social factors involved in the discovery of gravitational waves: https://blogs.sciencemag.org/books/2017/03/28/harry-collins-...

Re: What's wrong with social science and how to fix it

#77

Post by leftyted »

crazygringo wrote: post #65

I've tried to understand this (obviously quite angry/ranty) article and cannot actually figure out what data it has. It seems to not be based on actual replication results, but predicted replication results? But then the first chart isn't even predictions from the market, but just the author's predictions? The author clearly has a real hatred for practices in the social sciences. But I don't see any actual proof of t…

It's based on this: https://www.replicationmarkets.com, which is linked in the first paragraph of the article.

Per https://www.replicationmarkets.com/index.php/rules, volunteers are predicting whether 3000 social science papers are replicable. According to the rules, of those 3000 papers, ~5% will be resolved (i.e. attempts will be made to replicate). According to the article, 175 will be resolved. It's unclear to me who exactly will do that work but I would guess it's people behind replications markets dot com (they are funded by DARPA). The rules say that no one knows ahead of time which papers will be resolved so I assume the ~5% (or 175) will be chosen by random.

The data in the article seems to be based on what the forecasters predicted, not which papers actually replicated (that work hasn't been done yet...or at least hasn't been made public). The author of the article is assuming that the forecasters are accurate. To back up this assumption, he cites previous studies showing that markets are good at this kind of thing.

The tone is ranty but, by participating in the markets, the author is putting his money where his mouth is.

Re: What's wrong with social science and how to fix it

#78

Post by brandmeyer »

crazygringo wrote: post #65

I've tried to understand this (obviously quite angry/ranty) article and cannot actually figure out what data it has. It seems to not be based on actual replication results, but predicted replication results? But then the first chart isn't even predictions from the market, but just the author's predictions? The author clearly has a real hatred for practices in the social sciences. But I don't see any actual proof of t…

I think you're right. Take a look at the before/after curves for "this is what the predictions look like after the papers".

The before curves are gaussian+ distributed and pessimistic, but the after curves are all distinctly bimodal (or worse). This suggests that some population of the participants were broadly pessimized by their surveys and another population was broadly optimized by their surveys.

This could instead be a measurement of how people's trust in science is predicated on how well it matches their own prior beliefs.

+ A sharper eye shows they aren't quite bimodal in the prior belief. Even in those cases, the separation between the modes gets much wider.

Re: What's wrong with social science and how to fix it

#79

Post by boomboomsubban »

textman wrote: post #69
goatinaboat wrote: post #57

Earlier quoted context omitted.

You basically need to pay homage to the "ground breakers" who claimed that turf first, even if they did it via fraud. Even referring to it as “science” is fraudulent. Testable theories and repeatable outcomes, anyone? Time this whole field was defunded.

Might be that I am older than most of you, but when I was in high school the term was "social studies" then later it evolved to "social science" so the university crowd could legitimize it to get more funding. there obviously is no science in it.

"Social studies" refers to a variety of different social science and humanities subjects taught in public education. The term comes from US lawmakers around 1900.

"Social sciences" are generally considered the "science of society," and the term comes from philosophers in the mid-1800's.

Re: What's wrong with social science and how to fix it

#80

Post by liability »

pa7x1 wrote: post #59
closeparen wrote: post #41

Earlier quoted context omitted.

The physical sciences have no less "old white man conjecture" in their history.

They have but is not taught as science. Physics 101 does not study Aristotles ideas as part of physics. Not because they are wrong but because they are not scientific. They start with Galileo and Newton.

The first chemistry lecture I ever had started with something to the effect of "Chemistry emerged from alchemy, and we'll not speak of this again!"
Post reply on HN