Causal inference as a blind spot of data scientists

A/b experiments are definitely a gold standard as they provide true causality measurement (if implemented correctly). However, they are often expensive to run: need to implement the feature in question (which is less than 50% going to work) and then collect data for 1-4 weeks before being able to make the decision. As a result only a small number of business decisions today rely on a/b tests. Observational causal inference can help bring causality into many of the remaining decisions, which need to be made quicker or cheaper.

> In ML, you have a clear mechanism to check estimation/prediction through holdout approaches.

To be clear, you can overfit while your validation loss does not decrease. If your train and test data are too similar then no holdout will help you measure generalization. You have to remember that datasets are proxies for the thing you're actually trying to model, they are not the thing you are modeling themselves. You can usually see this when testing on in class but out of train/test distribution data (e.g. data from someone else).

You have to be careful because there are a lot of small and non-obvious things that can fuck up statistics. There's a lot of aggregation "paradoxes" (Simpsons, Berkson's), and all kinds of things that can creep in. This is more perilous the bigger your model too. The story of the Monte Hall problem is a great example of how easy it is to get the wrong answer while it seems like you're doing all the right steps.

For the article, the author is far too handwavy with causal inference. The reason we tend not to do it is because it is fucking hard and it scales poorly. Models like Autoregressive (careful here) and Normalizing Flows can do causal inference (and causal discovery) fwiw (essentially you need explicit density models with tractable densities: referring to Goodfellow's taxonomy). But things get funky as you get a lot of variables because there are indistinguishable causal graphs (see Hyvarien and Pajunen). Then there's also the issues with the types of causalities (see Judea's Ladder) and counterfactual inference is FUCKING HARD but the author just acts like it's no big deal. Then he starts conflating it with weaker forms of causal inference. Correlation is the weakest form of causation, despite our often chanted saying of "correlation does not equate to causation" (which is still true, it's just in the class and the saying is more getting at confounding variable). This very much does not scale. Similarly discovery won't scale as you have to permute so many variables in the graph. The curse of dimensionality hits causal analysis HARD.

Pearl’s framework isn’t really distinct from SEM as I understanding it. SEM is really just one tool to achieve the sort of adjustments that Pearl describes to make causal inferences from observational data.

> How to know if model 1 or model 2 is more "real" or, for my lack of a better term, more useful and reflective of reality?

I just want to second MJ's points here. You have to remember that 1) all models are wrong and 2) it's models all the way down. Your data is a model: it models the real world distribution, what we might call the target distribution, which is likely intractable and often very different from your data in various conditions. Your metrics are models: obviously given the previous point, but not as obvious from the point that even with perfect data these are still models. Your metrics all have limitations and you must be careful to clearly understand what they are measuring, rather than what you think they are. This is an issue of alignment and the vast majority of people do not consider precisely what their metrics mean and instead rely on the general consensus (great ML example: FID does not measure fidelity, it is distance measurement of distributions. But you shouldn't stop there, that's the start). These get especially fuzzy in higher dimensions where geometries are highly non-intuitive. It is best to remember that metrics are guides and not targets (Goodhart).

> There should be some way to determine a model's likelihood of structure beyond "trust me, it works!" If there is, I'm trying to understand it!

I mean we can use likelihood ;) if we model density of course. But that's not the likelihood that your model is the correct model, it is the likelihood that given the data that you have that your model's parameterization can reasonably model the sampling distribution of data. These are subtly different, the difference is from above. And then we gotta know if you're actually operating on the right number of dimensions. Are you approximating PCA like a typical VAE? Is the bottleneck enough for proper parameterization? Is your data in sufficient dimensionality? Does the fucking manifold hypothesis even hold for your data? What about the distribution assumption? IID? And don't get me started on indistinguishablity in large causal graphs (references in another comment).

So rather in practice it is just best to try to make a model that is robust to your data but always maintain suspicion of it. After all, all models are wrong and you're trying to model data, not have a model of data.

Evaluation is fucking hard (it is far too easy to make mistakes)

I love this comment to bits. Thanks, from a fellow applied researcher embedded in the tech world.