Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Causal inference as a blind spot of data scientists

dzidas.com

Post reply on HN
Print view View original

51–60 of 106 posts

Re: Causal inference as a blind spot of data scientists

#51

Post by muraiki »

kjkjadksj wrote: post #13
riedel wrote: post #7

Earlier quoted context omitted.

Actually causal inference is also really hard to benchmark. My colleague started an effort to be actually able to reproduce and compare results. Also the algorithms often do not scale too well. Everytime we wanted to use this for real data it is just a little bit too much effort and the results are not conclusive because it is hard to verify huge graphs. My colleague e.g. wanted to apply it explain risk confounders i…

You don’t need to look at a graph at all though, right? There are plenty of tests that can help you identify factors that could be significantly affecting your distribution

“Graph” here means the directed acyclic graph encoding the causal relationships, not a chart of a distribution.

Re: Causal inference as a blind spot of data scientists

#52

Post by jiggawatts »

mikpanko wrote: post #21
bertil wrote: post #2

The main reason for not using causal inference is not because data scientists don’t know about the different approaches or can’t imagine something equivalent (a lot of reinvention); forecasting is one of the most common tasks, after all. The main reason is that they generally work for software companies where it’s easier and less susceptible to analyst influence to implement the suggested change and test it with a Ra…

A/b experiments are definitely a gold standard as they provide true causality measurement (if implemented correctly). However, they are often expensive to run: need to implement the feature in question (which is less than 50% going to work) and then collect data for 1-4 weeks before being able to make the decision. As a result only a small number of business decisions today rely on a/b tests. Observational causal inf…

The “gold standard” has failure modes that seem to be ignored.

E.g.: making UI elements jump around unpredictably after a page load may increase the number of ad clicks simply because users can’t reliably click on what they actually wanted.

I see A/B testing turning into a religion where it can’t be argued with. “The number went up! It must be good!”

Re: Causal inference as a blind spot of data scientists

#53

Post by JHonaker »

shahbazac wrote: post #48

I’ve tried to understand causal inference several times and failed. Tutorials seem unnecessarily long winded. I wish authors would give simple, to the point examples. Say I have a simple table of outdoor temperatures and ice cream sales. What can the machinery of causal inference do for me in this situation? If it doesn’t apply here, what do I need to add to my dataset to make it appropriate for causal inference? Mor…

> Say I have a simple table of outdoor temperatures and ice cream sales.

You have more than that! You have knowledge about the world!

> What can the machinery of causal inference do for me in this situation?

Well, (I’m being purposefully pedantic here) you haven’t really asked a question yet. The first thing it can do is help you while you’re formulating one. It can answer questions like, “how can I anticipate how things I have and havent measured will the estimates I’m interested in/making?”

> If it doesn’t apply here, what do I need to add to my dataset to make it appropriate for causal inference? More columns of data? Explicit assumptions?

The first thing you need to do is articulate what you’re actually interested in. Then you need to be explicit about the causal relationships between things relevant to those questions. The big thing (to me) is that particular causal structures have testable conditional independence structures and by assessing these, you can build evidence for or against particular diagrams of the context.

Re: Causal inference as a blind spot of data scientists

#54

Post by tuukkah »

shahbazac wrote: post #48

I’ve tried to understand causal inference several times and failed. Tutorials seem unnecessarily long winded. I wish authors would give simple, to the point examples. Say I have a simple table of outdoor temperatures and ice cream sales. What can the machinery of causal inference do for me in this situation? If it doesn’t apply here, what do I need to add to my dataset to make it appropriate for causal inference? Mor…

TL;DR: Causal inference is a complex topic, not a simple tool.

How's the ice cream example better than the sugary snacks example given in the article?

Here's the part about needing to add more columns to the data:

> When dealing with a causal question, it’s crucial to include variables known as confounders. These are variables that can influence both the treatment and the outcome. By including confounding variables, we can better isolate and estimate the true causal effect of the treatment. Failing to add or account for confounding variables may lead to incorrect estimates.

Re: Causal inference as a blind spot of data scientists

#55

Post by bertil »

lr1970 wrote: post #47
mikpanko wrote: post #21

Earlier quoted context omitted.

A/b experiments are definitely a gold standard as they provide true causality measurement (if implemented correctly). However, they are often expensive to run: need to implement the feature in question (which is less than 50% going to work) and then collect data for 1-4 weeks before being able to make the decision. As a result only a small number of business decisions today rely on a/b tests. Observational causal inf…

There can be a real ethical dilemma when applying A/B testing in medical setting. Placing someone with an incurable disease in a control group is condemning them to death while in treatment group they might have a chance. On the other hand, without a proper A/B testing methodology the drug efficacy cannot be established. So far no perfect solution to the dilemma has been found.

All the alternative methods require the same sacrifice. More importantly, most suggested treatments fail to cure deadly conditions or have major side effects or risks that are just as unethical to thrust upon people untested.

If you look at it properly, i.e. evaluate what should be your actions before the test (Do nothing, Impose untested treatment, Test with proper control to learn what to do with the majority of the population), the answer is rarely ambiguous.

There is a debate to be had on how much pre-clinical work to be done before clinical testing, but those are increasingly automated, cheap, and fast, so we often reach the point where a double-blind test is the next logical step.

The argument you present is based on either an unwarranted confidence in treatments, or information that wasn’t available when the decision had to be made.

Re: Causal inference as a blind spot of data scientists

#56

Post by satvikpendem »

shahbazac wrote: post #48

I’ve tried to understand causal inference several times and failed. Tutorials seem unnecessarily long winded. I wish authors would give simple, to the point examples. Say I have a simple table of outdoor temperatures and ice cream sales. What can the machinery of causal inference do for me in this situation? If it doesn’t apply here, what do I need to add to my dataset to make it appropriate for causal inference? Mor…

Judea Pearl's The Book Of Why gives you more practical and easy to understand examples, I recommend that.

Re: Causal inference as a blind spot of data scientists

#58

Post by bertil »

jiggawatts wrote: post #52
mikpanko wrote: post #21

Earlier quoted context omitted.

A/b experiments are definitely a gold standard as they provide true causality measurement (if implemented correctly). However, they are often expensive to run: need to implement the feature in question (which is less than 50% going to work) and then collect data for 1-4 weeks before being able to make the decision. As a result only a small number of business decisions today rely on a/b tests. Observational causal inf…

The “gold standard” has failure modes that seem to be ignored. E.g.: making UI elements jump around unpredictably after a page load may increase the number of ad clicks simply because users can’t reliably click on what they actually wanted. I see A/B testing turning into a religion where it can’t be argued with. “The number went up! It must be good!”

That’s generally because the metrics you are looking at do not represent what users care about. It’s different than the testing methodology, often overlooked, and a lot more important.

I’ve argued that A/B testing training should focus on that skill a lot more than Welch’s theory, but I had to record my own classes for that to happen.

Re: Causal inference as a blind spot of data scientists

#59

Post by KRAKRISMOTT »

tuukkah wrote: post #54
shahbazac wrote: post #48

I’ve tried to understand causal inference several times and failed. Tutorials seem unnecessarily long winded. I wish authors would give simple, to the point examples. Say I have a simple table of outdoor temperatures and ice cream sales. What can the machinery of causal inference do for me in this situation? If it doesn’t apply here, what do I need to add to my dataset to make it appropriate for causal inference? Mor…

TL;DR: Causal inference is a complex topic, not a simple tool. How's the ice cream example better than the sugary snacks example given in the article? Here's the part about needing to add more columns to the data: > When dealing with a causal question, it’s crucial to include variables known as confounders. These are variables that can influence both the treatment and the outcome. By including confounding variables,…

> TL;DR: Causal inference is a complex topic, not a simple tool.

The simple version using graphical models and joint probabilities isn't difficult to explain or teach. The issue is that to do anything useful with it at scale you either need MCMC or variational inference and that's an entirely different bag of worms all together. For medical datasets you rarely have "scale", instead you have very few sample cases and a large expert model (the doctor/specialist).

Re: Causal inference as a blind spot of data scientists

#60

Post by nl »

tuukkah wrote: post #54
shahbazac wrote: post #48

I’ve tried to understand causal inference several times and failed. Tutorials seem unnecessarily long winded. I wish authors would give simple, to the point examples. Say I have a simple table of outdoor temperatures and ice cream sales. What can the machinery of causal inference do for me in this situation? If it doesn’t apply here, what do I need to add to my dataset to make it appropriate for causal inference? Mor…

TL;DR: Causal inference is a complex topic, not a simple tool. How's the ice cream example better than the sugary snacks example given in the article? Here's the part about needing to add more columns to the data: > When dealing with a causal question, it’s crucial to include variables known as confounders. These are variables that can influence both the treatment and the outcome. By including confounding variables,…

> How's the ice cream example better than the sugary snacks example given in the article?

Not the OP, but because that fails to explain how the basic hypothetical example works(!)

You want to know how much your sales would be in a parallel world where kids were stuck with bland snacks compared to your sweet treats. This is where causal inference steps in to provide the solution. (nice graph follows)

So how is that done?

Post reply on HN