We built some causal discovery and inference features with graph visualization in Kanaries RATH: https://docs.kanaries.net/rath/discover-causals/causal-analy...
It's also open-sourced. Welcome to have a try.
Causal inference as a blind spot of data scientists
91–100 of 106 posts
Re: Causal inference as a blind spot of data scientists
#92Earlier quoted context omitted.
When looking at admissions rates across the entire university, women are less likely to be accepted than men. But when (in this example) you break that down into departments, every department favors women over men. If every department favored women then the entire university would also favor women. Parity is guaranteed in that scenario. What happened in the Berkeley case is that not every department favored women, an…
Yes, apologies, what I meant by "favored" was that in every department, women applicants were more likely to get an admission than men. But I'm pretty sure the admission rate can still be lower for women overall than men overall, using exactly the same scenario you described. If the sociology department admits 10 percent of applicants and the physics department admits 90, it seems very easy for gender bias in applica…
I get your point now. You're quite right that you can construct scenarios that arbitrarily favor men in the aggregate but women in specific departments, given the right ratio of applicants.
Re: Causal inference as a blind spot of data scientists
#93I’ve tried to understand causal inference several times and failed. Tutorials seem unnecessarily long winded. I wish authors would give simple, to the point examples. Say I have a simple table of outdoor temperatures and ice cream sales. What can the machinery of causal inference do for me in this situation? If it doesn’t apply here, what do I need to add to my dataset to make it appropriate for causal inference? Mor…
> Or do I need to provide that graph to this function? You need to do that, and the math can help you measure how much each arrow contributes. The idea that you need to provide your model of the world is strangely not a key part of most introductions, but it’s crucial. > outdoor temperatures and ice cream sales That’s too simple: a simple regression can handle that. Causal inference can handle cases with three variab…
> > outdoor temperatures and ice cream sales
> That’s too simple: a simple regression can handle that.
Not quite. Regression by itself will not answer the causal (or equivalently, the counterfactual) question.
I strongly suspect you already know this and was elaborating on a related point. But just for the sake of exposition, let me add a few words for the HN audience at large.
Let me give an example. In an email corpus, mails that begin with "Honey sweetheart," will likely have a higher than baseline open rate. A regression on word features will latch on to that. However, if your regular employer starts leading with "Honey sweetheart" that will not increase the open rate of corporate communications.
Causal or counterfactual estimation is fundamentally about how a dependent variable responds to interventional changes in a causal variable. Regression and relatedly, conditional probabilities are about 'filtering' the population on some predicate.
An email corpus when filtered upon the opening phrase "Honey sweetheart" may have disproportionately high email open rates, but that does not mean that adding or adopting such a leading phrase will increase the open rate.
Similarly, regressing dark hair as a feature against skin cancer propensity will catch an anti-correlation effect. Dyeing blonde hair dark will not reduce melanoma propensity.
Re: Causal inference as a blind spot of data scientists
#94In Corporate and Medical data science fields, people begin to accept causal inference. It is difficult, as the subject is still in flux and under development. I am aware of three reputable causal inference frameworks: 1. Judea Pearl's framework, which dominates in CS and AI circles 2. Neyman-Rubin causal model: https://en.wikipedia.org/wiki/Rubin_causal_model 3. Structural equation modelling: https://en.wikipedia.org…
Social Scientist here. It is thriving under the name Qualitative comparative analysis for a quarter of a century. This is a good paper for more on the epistemological foundations: https://doi.org/10.1177/1098214016673902
Re: Causal inference as a blind spot of data scientists
#95Earlier quoted context omitted.
A/b experiments are definitely a gold standard as they provide true causality measurement (if implemented correctly). However, they are often expensive to run: need to implement the feature in question (which is less than 50% going to work) and then collect data for 1-4 weeks before being able to make the decision. As a result only a small number of business decisions today rely on a/b tests. Observational causal inf…
There can be a real ethical dilemma when applying A/B testing in medical setting. Placing someone with an incurable disease in a control group is condemning them to death while in treatment group they might have a chance. On the other hand, without a proper A/B testing methodology the drug efficacy cannot be established. So far no perfect solution to the dilemma has been found.
You can end the trial early when it’s clear the treatment is working. This just happened last week with Ozempic for diabetes caused kidney disease.
https://www.wxyz.com/news/health/ask-dr-nandi/novo-nordisk-e...
Re: Causal inference as a blind spot of data scientists
#96Earlier quoted context omitted.
I love this comment to bits. Thanks, from a fellow applied researcher embedded in the tech world.
I always love to find/know there are others in the tech world that care about the nuance around evaluation math and not just benchmarks. Often it feels like I'm alone. So thank you!
Would love to connect. Feel free to reach out, msg box address available if a bit mangled in my profile.
Re: Causal inference as a blind spot of data scientists
#97It's not that hard. if the causality cannot make sense logically or plausibly, then you can reasonably reject it . no reasonable person would ever get the umbrella puddles thing confused.
> It's not that hard. if the causality cannot make sense logically or plausibly, then you can reasonably reject it . no reasonable person would ever get the umbrella puddles thing confused. It's an illustrative example, taking it literally is missing the point. The reason you know it doesn't make sense for umbrellas to cause rain is that you already have an applicable causal model. The situations where you need to do…
This is not true. These causal methods generally require you to have a pre established framework for how the thing works. If you cannot supply additional variables that you, with your domain knowledge, know cover the confounding elements, it won’t do anything.
it’s Mathematical soup for trying to normalize out the effects of other variables to see what remains and calling it “causal”.
Re: Causal inference as a blind spot of data scientists
#98Earlier quoted context omitted.
I realize this is nitpicking a minor point in your comment, but I don't agree with your characterization of RCTs in medical research as being primarily constrained by laws and regulations. Any time I've discussed research on human subjects with doctors doing that research, the discussion of what is and is not an acceptable experiment has always been primarily driven by the risks of harm to the people involved in the…
That's a very fair point. I didn't mean to suggest that harm to the patients or subjects was not the overriding factor, nor that bio, pharma, and other medical fields never do RCTs. But there are a slew of laws and requirements around _how_ to run an RCT across the world of bio-related work, esp as it becomes a product. From marketing to manufacture to packaging, there are strict limits around where variation is allo…
Yes, that's true, legal requirements definitely become much more of a factor the closer you get from research to product.
Re: Causal inference as a blind spot of data scientists
#99Earlier quoted context omitted.
A/b experiments are definitely a gold standard as they provide true causality measurement (if implemented correctly). However, they are often expensive to run: need to implement the feature in question (which is less than 50% going to work) and then collect data for 1-4 weeks before being able to make the decision. As a result only a small number of business decisions today rely on a/b tests. Observational causal inf…
There can be a real ethical dilemma when applying A/B testing in medical setting. Placing someone with an incurable disease in a control group is condemning them to death while in treatment group they might have a chance. On the other hand, without a proper A/B testing methodology the drug efficacy cannot be established. So far no perfect solution to the dilemma has been found.
Most therapeutic trials are nowadays "Intent to treat". So subject would receive either standardized tx or experimental tx in th e randomization. Many of them also have crossovers such that when measurable (as defined by the protocol) benefit is seen, standard tx based subjects can be moved over to the experimental arm
Re: Causal inference as a blind spot of data scientists
#100Earlier quoted context omitted.
> Or do I need to provide that graph to this function? You need to do that, and the math can help you measure how much each arrow contributes. The idea that you need to provide your model of the world is strangely not a key part of most introductions, but it’s crucial. > outdoor temperatures and ice cream sales That’s too simple: a simple regression can handle that. Causal inference can handle cases with three variab…
> > outdoor temperatures and ice cream sales > That’s too simple: a simple regression can handle that. Not quite. Regression by itself will not answer the causal (or equivalently, the counterfactual) question. I strongly suspect you already know this and was elaborating on a related point. But just for the sake of exposition, let me add a few words for the HN audience at large. Let me give an example. In an email cor…
Your model needs to introduce a third piece of information: whether an email is a corporate communication—or a deliberate intervention.