Live data from Hacker News

Causal inference as a blind spot of data scientists

dzidas.com

71–80 of 106 posts

Re: Causal inference as a blind spot of data scientists

#71

> The DoubleML method is founded on machine learning modeling and consists of two key steps. First, we build a model that predicts the treatment variable based on the input variables . Then, we create a separate model that predicts the outcome variable using the same set of input variables . Subsequently, we calculate the residuals from the former model and regress them against the residuals from the latter model. An…

Not sure what point you are trying to make here. Double ML is a valid approach for debiasing confounding effects.

Re: Causal inference as a blind spot of data scientists

#72

It's not that hard. if the causality cannot make sense logically or plausibly, then you can reasonably reject it . no reasonable person would ever get the umbrella puddles thing confused.

You should open a epidemiological journal these days. Half the papers are either as bad as "umbrellas causes puddles" or obviously confounded with socio-economic status.

Re: Causal inference as a blind spot of data scientists

#73

I’ve tried to understand causal inference several times and failed. Tutorials seem unnecessarily long winded. I wish authors would give simple, to the point examples. Say I have a simple table of outdoor temperatures and ice cream sales. What can the machinery of causal inference do for me in this situation? If it doesn’t apply here, what do I need to add to my dataset to make it appropriate for causal inference? Mor…

[deleted]

Re: Causal inference as a blind spot of data scientists

#74
post #3

For a hands on introduction to Causality, I would recommend “Causal Inference in Python” by M. Facure https://amzn.to/46byWnl Well written and to the point. I also have a series of blog posts on the topic: https://github.com/DataForScience/Causality where I work through Pearls Primer: https://amzn.to/3gsFlkO

Thank you for sharing

Re: Causal inference as a blind spot of data scientists

#75
post #58

Earlier quoted context omitted.

The “gold standard” has failure modes that seem to be ignored. E.g.: making UI elements jump around unpredictably after a page load may increase the number of ad clicks simply because users can’t reliably click on what they actually wanted. I see A/B testing turning into a religion where it can’t be argued with. “The number went up! It must be good!”

That’s generally because the metrics you are looking at do not represent what users care about. It’s different than the testing methodology, often overlooked, and a lot more important. I’ve argued that A/B testing training should focus on that skill a lot more than Welch’s theory, but I had to record my own classes for that to happen.

But those metrics are hard to move, so you target secondary metrics.

The problem with that strategy becomes obvious when you spell out the consequences: measurably improving the product is hard, so you measure something else and hope you get product improvements.

Re: Causal inference as a blind spot of data scientists

#76

For what it’s worth, my undergraduate was in Economics with an emphasis in econometrics and this article touched on probably 80% of the curriculum. The only problem is by the time I graduated I was somewhat disillusioned with most causal inference methods. It takes a perfect storm natural experiment to get any good results. Plus every 5 years a paper comes out that refutes all previous papers that use whatever method…

just use propensity scores + ipw and you have the same thing as a rct. :)

From my experience propensity scores + ipw really doesn't get you far in practice. Propensity scoring models rarely balance all the covariates well (more often, one or two are marginally better and some may be worse than before). On top of that, IPW either assumes you don't have any cases of extreme imbalance, or, if you do you end up trimming weights to avoid adding additional variance, but in some cases you do even with trimmed weights..

Re: Causal inference as a blind spot of data scientists

#77

For what it’s worth, my undergraduate was in Economics with an emphasis in econometrics and this article touched on probably 80% of the curriculum. The only problem is by the time I graduated I was somewhat disillusioned with most causal inference methods. It takes a perfect storm natural experiment to get any good results. Plus every 5 years a paper comes out that refutes all previous papers that use whatever method…

Yeah, the only common theme I see in causal inference research is that every method and analysis eventually succumbs to a more thorough analysis that uncovers serious issues in the assumptions.

Take for instance the running example of catholic schoolings effect on test scores used by the boook Counterfactuals and Causal Inference. Subsequent chapter re-treat this example with increasingly sophisticated techniques and more complex assumptions about causal mechanisms, and each time they uncover a flaw in the analysis using techniques from previous chapters.

My lesson from this: outcomes causal inference is very dependent on assumptions and methodologies, of which the options are many. This is a great setting for publishing new research, but its the opposite of what you want in an industry setting where the bias is/should be towards methods that are relatively quick to test and validate and put in production.

I see researchers in large tech companies pushing for causal methodologies, but I'm not convinced they're doing anything particularly useful since I have yet to see convincing validation on production data of their methods that show they're better than simpler alternatives which will tend to be more robust.

Re: Causal inference as a blind spot of data scientists

#78
A co-worker pointed me to this e-book, which I thought did a great job of presenting he concepts in a relatable and applied way:

https://matheusfacure.github.io/python-causality-handbook/la...

But I agree with other comments here, at the end of the day it seems like causal analysis often boils down to whether you trust the analyst and/or their techniques since it is hard to validate the results.

Re: Causal inference as a blind spot of data scientists

#79

For what it’s worth, my undergraduate was in Economics with an emphasis in econometrics and this article touched on probably 80% of the curriculum. The only problem is by the time I graduated I was somewhat disillusioned with most causal inference methods. It takes a perfect storm natural experiment to get any good results. Plus every 5 years a paper comes out that refutes all previous papers that use whatever method…

Yeah, the only common theme I see in causal inference research is that every method and analysis eventually succumbs to a more thorough analysis that uncovers serious issues in the assumptions. Take for instance the running example of catholic schoolings effect on test scores used by the boook Counterfactuals and Causal Inference. Subsequent chapter re-treat this example with increasingly sophisticated techniques and…

> My lesson from this: outcomes causal inference is very dependent on assumptions and methodologies, of which the options are many.

This seems like a natural feature of any sensitive method, not sure why this is something to complain about. If you want your model to always give the answer you expected you don't actually have to bother collecting data in the first place, just write the analysis the way pundits do.

Re: Causal inference as a blind spot of data scientists

#80
post #47

Earlier quoted context omitted.

A/b experiments are definitely a gold standard as they provide true causality measurement (if implemented correctly). However, they are often expensive to run: need to implement the feature in question (which is less than 50% going to work) and then collect data for 1-4 weeks before being able to make the decision. As a result only a small number of business decisions today rely on a/b tests. Observational causal inf…

There can be a real ethical dilemma when applying A/B testing in medical setting. Placing someone with an incurable disease in a control group is condemning them to death while in treatment group they might have a chance. On the other hand, without a proper A/B testing methodology the drug efficacy cannot be established. So far no perfect solution to the dilemma has been found.

It's not really an ethical dilemma until you know it works, and then usually if the evidence is strong enough they'll cut the trial early.
Post reply on HN