Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Causal inference as a blind spot of data scientists

dzidas.com

Post reply on HN
Print view View original

41–50 of 106 posts

Re: Causal inference as a blind spot of data scientists

#41

Post by bertil »

mikpanko wrote: post #36
bertil wrote: post #24

Earlier quoted context omitted.

Causal inference is useful, but it's neither quicker nor cheaper.

Agree that it is hard today. A person you might know is trying to prove that is doesn’t have to be: https://www.motifanalytics.com/blog/bringing-more-causality-... . We’d love to chat more with you on the topic - feel free to hit Sean or me on LinkedIn.

I am a big fan of what Sean and you are trying to do–I wrote up a chapter about it this weekend, actually. I’m worried that you both have worked for companies where a lot of work has been done to identify relevant dimensions (metrics and categories) and automate causality (or rather: estimating factors on a pre-existing causal graph because that’s the slight of hands the word “causality” does) made sense once you’ve reached that level of maturity.

But to reach that point, before having relevant dimensions, there has to be a lot of work, generally motivated by disappointing experiments. “Why didn’t that work?” is often answered by “Because our goal is too remote from our actions—here’s a better proxy” or “Because this change only makes sense to 8% of our users, here’s how we can split them.”

I’m worried that too many people will think the tool itself is enough and not a complement to the maturity in understanding a company’s user. This ‘solutionism’ is widespread among Data tools: https://www.linkedin.com/posts/bertilhatt_the-potential-gap-...

Re: Causal inference as a blind spot of data scientists

#42

Post by tmoertel »

Interestingly, recent research suggest that you can make better decisions by combining experimental and observational data than by using either alone:

https://ftp.cs.ucla.edu/pub/stat_ser/r513.pdf

> Abstract: Personalized decision making targets the behavior of a specific individual, while population-based decision making concerns a sub-population resembling that individual. This paper clarifies the distinction between the two and explains why the former leads to more informed decisions. We further show that by combining experimental and observational studies we can obtain valuable information about individual behavior and, consequently, improve decisions over those obtained from experimental studies alone.

Re: Causal inference as a blind spot of data scientists

#43

Post by mwexler »

rcthompson wrote: post #16
mwexler wrote: post #9

I 100% agree with this blind spot. Most data science coursework avoids the very thing making it a science: the explanation of what change causes what effect. I've been surprised that year after year, programs at so many "Schools of Data Science" keep gliding over this area, perhaps alluding to it in an early stats course if at all. It's an important part of validating that your data-driven output or decision is actua…

I realize this is nitpicking a minor point in your comment, but I don't agree with your characterization of RCTs in medical research as being primarily constrained by laws and regulations. Any time I've discussed research on human subjects with doctors doing that research, the discussion of what is and is not an acceptable experiment has always been primarily driven by the risks of harm to the people involved in the…

That's a very fair point. I didn't mean to suggest that harm to the patients or subjects was not the overriding factor, nor that bio, pharma, and other medical fields never do RCTs.

But there are a slew of laws and requirements around _how_ to run an RCT across the world of bio-related work, esp as it becomes a product. From marketing to manufacture to packaging, there are strict limits around where variation is allowed, at least anything involving the FDA in the US. (Some would say too many regs, others say not enough).

And in those cases, having a wider collection of ways to impute cause would be great.

Re: Causal inference as a blind spot of data scientists

#44

Post by huitzitziltzin »

g42gregory wrote: post #22

In Corporate and Medical data science fields, people begin to accept causal inference. It is difficult, as the subject is still in flux and under development. I am aware of three reputable causal inference frameworks: 1. Judea Pearl's framework, which dominates in CS and AI circles 2. Neyman-Rubin causal model: https://en.wikipedia.org/wiki/Rubin_causal_model 3. Structural equation modelling: https://en.wikipedia.org…

> In Social Sciences, on the other hand, causal inference is being completely willfully ignored. Why? Causal inference is an obstacle to making a preconceived conclusions based on pure correlations: something correlates with something, therefore ... invest large sums of money, change laws in our favor, etc... This works for both sides. Sadly, I don't think this could be fixed.

This remark is totally ignorant of the reality in the social sciences. Certainly in economics (which I know well) this hasn't described the reality of empirical work for more than 30 years. Political Science and Sociology are increasingly concerned with causal methods as well.

Medicine on the other hand is the opposite. Medical journals generally publish correlations when they aren't publishing experiments.

Re: Causal inference as a blind spot of data scientists

#45

Post by quotient »

Anon84 wrote: post #3

For a hands on introduction to Causality, I would recommend “Causal Inference in Python” by M. Facure https://amzn.to/46byWnl Well written and to the point. I also have a series of blog posts on the topic: https://github.com/DataForScience/Causality where I work through Pearls Primer: https://amzn.to/3gsFlkO

The Facure text is good, can confirm

Re: Causal inference as a blind spot of data scientists

#46

Post by mikpanko »

bertil wrote: post #41
mikpanko wrote: post #36

Earlier quoted context omitted.

Agree that it is hard today. A person you might know is trying to prove that is doesn’t have to be: https://www.motifanalytics.com/blog/bringing-more-causality-... . We’d love to chat more with you on the topic - feel free to hit Sean or me on LinkedIn.

I am a big fan of what Sean and you are trying to do–I wrote up a chapter about it this weekend, actually. I’m worried that you both have worked for companies where a lot of work has been done to identify relevant dimensions (metrics and categories) and automate causality (or rather: estimating factors on a pre-existing causal graph because that’s the slight of hands the word “causality” does) made sense once you’ve…

Thank you for clarifying.

Reading some of your posts I think we agree more than disagree. A big difference from most new analytics tools you see today is that we don't want to provide a magic "solution" (which is bound to over-promise and under-deliver) but rather a generic tool to quickly define and try out different business categories on the data.

Followed you on LinkedIn for more in-depth takes.

Re: Causal inference as a blind spot of data scientists

#47

Post by lr1970 »

mikpanko wrote: post #21
bertil wrote: post #2

The main reason for not using causal inference is not because data scientists don’t know about the different approaches or can’t imagine something equivalent (a lot of reinvention); forecasting is one of the most common tasks, after all. The main reason is that they generally work for software companies where it’s easier and less susceptible to analyst influence to implement the suggested change and test it with a Ra…

A/b experiments are definitely a gold standard as they provide true causality measurement (if implemented correctly). However, they are often expensive to run: need to implement the feature in question (which is less than 50% going to work) and then collect data for 1-4 weeks before being able to make the decision. As a result only a small number of business decisions today rely on a/b tests. Observational causal inf…

There can be a real ethical dilemma when applying A/B testing in medical setting. Placing someone with an incurable disease in a control group is condemning them to death while in treatment group they might have a chance. On the other hand, without a proper A/B testing methodology the drug efficacy cannot be established. So far no perfect solution to the dilemma has been found.

Re: Causal inference as a blind spot of data scientists

#48

Post by shahbazac »

I’ve tried to understand causal inference several times and failed. Tutorials seem unnecessarily long winded. I wish authors would give simple, to the point examples.

Say I have a simple table of outdoor temperatures and ice cream sales.

What can the machinery of causal inference do for me in this situation?

If it doesn’t apply here, what do I need to add to my dataset to make it appropriate for causal inference? More columns of data? Explicit assumptions?

If I can use causal inference, what can it tell me? If I think of it as a function CA(data), can it tell me if the relationship is actually causal? Can it tell me the direction of the relationship? If there were more columns, could it return a graph of causal relationships and their strength? Or do I need to provide that graph to this function?

I know a wet pavement can be caused by rain or spilled water or that an alarm can go off due to an earthquake or a burglary. I have common sense. I also understand the basics of graph traversal from comp sci classes.

How do I practically use causal inference?

To the authors of future articles on this (or any technical tutorial), please explain the essence, the easy path, then the caveats and corner cases. Only then will abstract philosophizing make sense.

Re: Causal inference as a blind spot of data scientists

#49

Post by travisjungroth »

mikpanko wrote: post #21
bertil wrote: post #2

The main reason for not using causal inference is not because data scientists don’t know about the different approaches or can’t imagine something equivalent (a lot of reinvention); forecasting is one of the most common tasks, after all. The main reason is that they generally work for software companies where it’s easier and less susceptible to analyst influence to implement the suggested change and test it with a Ra…

A/b experiments are definitely a gold standard as they provide true causality measurement (if implemented correctly). However, they are often expensive to run: need to implement the feature in question (which is less than 50% going to work) and then collect data for 1-4 weeks before being able to make the decision. As a result only a small number of business decisions today rely on a/b tests. Observational causal inf…

> As a result only a small number of business decisions today rely on a/b tests.

The default for all code changes at Netflix is they’re A/B tested.

Re: Causal inference as a blind spot of data scientists

#50

Post by atrettel »

mumblemumble wrote: post #6

I would argue it's more a blind spot of big data, which tends to tacitly imply just doing correlational studies on data that happens to be laying around. Most data scientists work for companies that don't really want to pay for controlled experiments outside of maybe letting the UI team do A/B tests. Natural experiments can be hard to come by in a business setting. And all of the wild mathematical gyrations that econ…

Exactly. This is the primary difference between observational and experimental studies (controlled experiments). Experimental studies control for the hypothesized mechanism as part of the experimental design, but observation studies do not or often cannot. Good data from controlled experiments is difficult, costly, and time-consuming to generate in general, and that often does not mesh with the notion of "big data". I think we are running into this problem more and more as we discover that our data sets really are superficial --- collections of a lot of data that is easy to collect rather than a representative sample of everything (especially in a controlled manner). Good data isn't cheap.
Post reply on HN