What does social science have to offer the data industry?

I think one way to think of this is, does a given field have any tools that, even if you disagreed with their values, you would still want access to? People who don't like the idea of natural selection, still want doctors to take into account the phenomenon of antibiotic-resistant infections in their treatment. People who dislike the values of the software industry, often still want access to computers to publish their essays, surf the internet, etc. People who dislike the analytic, anti-holistic orientation of the physical sciences, still want access to the technology made using that.

What is there in the social sciences that you would want access to, even if you did not share their values? I think we may live to see a day when there is something, but I'm not sure that right now there is (yet).

I suspect some topics which you might think are just "common sense" come from intensive research.

We’ve seen the effect of it in management over the past 25 years. Today a good manager is expected to approach a team, not by instructing them in what to do and when to do it, but rather by creating a shared meaning through group conversation. It’s more important when you manage people who produce by thinking and being creative, but even at the factory line, this softer approach is proving useful.

We haven’t yet applied this to big data. I’m often sold ML as the ability to predict the future, and to some extend that is true. If I look at all the alcoholic families in my municipality and compare their case history with big data gathered on a national level, I’ll certainly be able to predict how many of their children we’ll need to remove. I just can’t predict which ones because determinism doesn’t actually work on something that complex.

The more data we have the less we understand about causality, something I’ve learned from history. If you look at the Roman Empire without digging into it, chosing Christianity seem obvious, but if you really get all the data on their options and then try to figure out why they did like they did, you’ll have no clue. Another example is online advertising, I read a news paper that I’ve never seen a single add for, and I see a lot of adds for news papers. I’m often called by news paper salesmen as well, but not for the one I read. This is because it doesn’t suit my elaborate online profile. My profile tells the add agencies what I should read, but it doesn’t tell them why, and the difference is failing them.

If we really want ML and big data to be truely useful, I think we need to learn from the social sciences, because they work much more with the complicated science behind the why.

Google’s Deep Dream is a great way to visualize this. Given a source image that you repeatedly feed through an algorithm that attempts to parse and recreate the image, an unbiased algorithm would produce something similar to the original. Instead you get dogfishbirds and eyes everywhere — that’s the bias of the training set getting amplified.

I'm not necessarily suggesting that as a bad thing, I just wanted to clarify that it's actually very easy to come up with some very unpleasant data that is totally devoid of bias at all.