How to Think about Correlation?

If R-squared is 10% (and p-value is 0.01), does it mean the correlation is 10% better than random?

Correlations are a profound part of our universe.

When observing the universe, humans can never prove any facts about the universe. We can only establish correlations. Correlations between events that occur in our universe is the furthest "truth" we can establish about the universe short of a full on proof.

What this means is that nothing in the physical universe can be proven. Proof is the domain of maths and logic, correlations is the domain of science. Science cannot prove anything, it can only establish correlations and causations.

The reason this occurs is because at any time in the future one can observe an event that contradicts a hypothesis. You can hypothesize that all birds have wings and observe 2 trillion birds with wings but you never know when one day you'll observe a bird without wings disproving your entire hypothesis. That is why nothing can be proven, you can only correlate things through observation.

The other interesting part about correlation is what it isn't: Causation. People often talk about how correlation is not causation but people never talk about what causation is and how to establish it. If I can't empirically use correlation to establish causation how on earth is causation ever formally established? People rarely question this disconnect.

The fact is, causation is rarely formally established but a method does exist and it's subtle. If I observe that whenever Bob flicks a switch the light comes on then I established that the light coming on is correlated with Bob flipping the switch. This is as far as I can go with just observation. To establish causation I must make myself both an observer and an entity that is part of the system itself. I have to take control and flip the switch randomly and observe that when I don't flip the switch nothing happens and when I do flip the switch the lights come on.

By doing this I establish causation. To establish causation to higher and higher degrees I need to Cause (keyword) random events and make sure that a cause influences an effect AND absence of a cause and therefore absence of an effect occurs.

Also note that establishing causation is not proof. At any point in time in the future I can flip the switch and the light may not come on which is contradictory evidence for causation. Causation in the statistical sense is like correlation, you establish it to a degree of confidence but you can never Prove that A caused B.

R^2 of 10% means 10% of the variance in one variable is accounted for by the linear regression on the other variable. That is, the expectation of the conditional variance (the amount of variance left after using the information from the linear regression) is 90% of the unconditional variance.

A p-value of 0.01 means that under the null hypothesis (generally that the variables are independent), a result at least this extreme (in this case, with at least this large a correlation) would be seen 1% of the time.

Note in general that R^2 = 10% does not imply p = 0.01, nor the converse. They are largely unrelated measures. In principle one could have a small correlation with high significance, or a large correlation with low significance.

Note that there are ways to test causal hypotheses without intervening.

For example, suppose we wish to test the hypothesis that smoking causes lung cancer via the main mechanism of tar buildup in the lungs, against the alternative hypothesis that smoking is correlated with lung cancer because of a gene that predisposes people to both smoking and lung cancer (this example comes from Judea Pearl’s Book of Why).

If the former hypothesis is true, then we should see a correlation between smoking behavior and tar deposits, and we should also see a correlation between tar deposits and lung cancer even after controlling for smoking behavior. Composing the causal effects at each stage, we can then calculate the indirect causal effect of smoking on lung cancer. If this suffixes to explain the correlation, then that rules out the alternative genetic explanation for the correlation.

Of course, we can always propose ad-hoc further hypotheses that complicate the analysis: maybe the correlation between smoking and tar deposits is itself non-causative, etc. This only goes to show that scientific inquiry must be done with judgment and with expert domain knowledge, testing plausible hypotheses in good faith. But that’s true for detecting mere correlations too — we can always doubt our instruments, or claim the data is a statistical fluke. And it’s true in randomized controlled trials: it’s conceivable we didn’t properly randomize, etc.

This (causation as correlation) revelation was highlighted by Hume and this and other work by him had profound influence on Kant (famously awaking him from his "dogmatic slumbers") and scientists like Darwin and Einstein - the latter obviously in a more healthy scientific age when those at the forefront of physics were not so disdainful of philosophers.