Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Everything Is Correlated

gwern.net

Post reply on HN
Print view View original

31–40 of 56 posts

Re: Everything Is Correlated

#31

Post by dwaltrip »

nonbel wrote: post #26
gwern wrote: post #25

Earlier quoted context omitted.

> It's cheating, it's goes against experimental design analysis, and it does not differentiate between given data and data that was carefully collected. We have experimental design class for a reason. It helps us to be honest. Of course there are tons of pit falls many novice statisticians can do. Explicit sequential testing runs into exactly the same problem. The problem is, the null hypothesis is not true. So no ma…

This is mostly right, but you are still thinking of these rejections as "false positives" for some reason. They are real deviations from the null hypothesis ("true positives"). The problem is the user didn't test the null model they wanted, it is 100% user error.

Can you explain that last sentence? What is a valid null model if everything is correlated?

Re: Everything Is Correlated

#32

Post by nonbel »

dwaltrip wrote: post #31
nonbel wrote: post #26

Earlier quoted context omitted.

This is mostly right, but you are still thinking of these rejections as "false positives" for some reason. They are real deviations from the null hypothesis ("true positives"). The problem is the user didn't test the null model they wanted, it is 100% user error.

Can you explain that last sentence? What is a valid null model if everything is correlated?

A model of whatever process you think generated the data.

EDIT:

I guess I should say that the concept of testing a "null model" without interpreting the fit relative to other models is wrong to begin with. You need to use Bayes' rule and determine:

  p(H[0]|D) = p(H[0])p(D|H[0])/sum(p(H[0:n])*p(D|H[0:n]))
Lots of stuff wrong with what has been standard stats for the last 70 years, it literally amounts to stringing together a bunch of fallacies and makes no sense at all.

Re: Everything Is Correlated

#33

Post by RosanaAnaDana »

I think a major issue here is that, perhaps, there is a tendency to want to use statistics to decide what the 'truth' is, because it takes the onus of responsibility for making a mistake away from the interpreter. Its nice to be able to stand behind a p-value and not be accountable for whatever argument is being made. But the issue here, is that most any argument can be made in a large enough dataset, and a careful analyst will find significance.

This is of course the case only if one does not venture far from the principal assumptions of frequentism, most of which are routinely violated outside of almost every example except pure random number generation and fundamental quantum physics.

So a central issue that isn't addressed in STATS101 level hypothesis testing is the impact that the question has on the result. Its almost inevitable that people want to interpret a failure to reject as a positive result. But a p-value really doesn't tell you if its a useful result; but rather, your sample size is big enough to detect a difference.

Statistical significance is something that can be calculated. Practical significance is something that needs to be interpreted.

Re: Everything Is Correlated

#34

Post by nonbel »

Mistletoe wrote: post #17

This reminds me of the current omnigenic hypothesis about genes. That unexpectedly almost every gene seems to affect the expression of traits. https://www.quantamagazine.org/omnigenic-model-suggests-that... "Drawing on GWAS analyses of three diseases, they concluded that in the cell types that are relevant to a disease, it appears that not 15, not 100, but essentially all genes contribute to the condition. The author…

That is just a special case of the "everything is correlated" principle.

Re: Everything Is Correlated

#35

Post by nonbel »

gwern wrote: post #29
nonbel wrote: post #24

>" The fact that these variables are all typically linear or additive further implies that interactions between variables will be typically rare or small or both (implying that most such hits will be false positives, as interactions are far harder to detect than main effects)." Where does this "fact" come from? And if everything is correlated with everything else all these effects are true positives... Also, another…

> Finally, I didn't read through the whole thing. Does he claim to have found an exception to this rule at any point? Oakes 1975 points out that explicit randomized experiments, which test a useless intervention such as school reform, can be exceptions. (Oakes might not be quite right here, since surely even useless interventions have some non-zero effect, if only by wasting peoples' time & effort, but you might say…

Thanks,

How about this "fact": The fact that these variables are all typically linear or additive?

Re: Everything Is Correlated

#36

Post by nonbel »

dalore wrote: post #10
mojomark wrote: post #8

Earlier quoted context omitted.

Good discussion. On the flip side, in my data mining class the professor keeps saying ~"you may be able to find clusters in a data set, but often no true correlation exists." However, that's an absolute statement I just don't swallow. In my mind what I see is that if an unexplained correlation or non-correlation appears, it may be random (or true) or it could be the result of an unmeasured (hidden) variable. In your…

Perhaps they are trying to explain clustering illusion? The phenomenon that even random data will produce clusters. You can take that further and state random data WILL produce clusters. If you don't have clusters then your data is not random and some pattern is at play. This really tricks up our mind as our mind tries to find patterns everywhere. If you try and plot random dots you will usually put dots without clus…

>"The phenomenon that even random data will produce clusters."

You don't really mean "random", you mean i.i.d. You can have a statistical model where the probability of something happens is random, but not independent of the past values (eg, the next step a markov chain).

Re: Everything Is Correlated

#37

Post by Sniffnoy »

Question: Are these correlations typically transitive? That is to say, does it typically happen that in addition to everything having nonzero correlation with everything else, it additionally happens that the sign of the correlation between A and C is equal to the product of the signs of the correlations between A and B and between B and C?

Thorndike's dictum would suggest that this is so, at least in that particular domain. What about more generally?

Re: Everything Is Correlated

#38

Post by mercer »

seiferteric wrote: post #20
ajuc wrote: post #7

When the correlation is close to 0 it's often because of a feedback loop. For example - in economy with central bank trying to hit inflation target - interest rates and inflation will have near 0 correlation (interest rates change but inflation remains constant). That's because central bank adjusts interest rates to counter other variables so that inflation remains near the target. Other example (my favorite, it was…

I really like that example, but I am wondering if it would really be true? Real drivers would not maintain a perfect speed, but would instead work to maintain the average. If you looked closely at the speed, it would drift away from the average, then the peddle would move to return it to the average. So it would look a bit like an integral (The I in PID control) of the difference from the mean speed right?

Well, we're obviously talking about perfectly spherical drivers (good point though).

Re: Everything Is Correlated

#39

Post by seiferteric »

mercer wrote: post #38
seiferteric wrote: post #20

Earlier quoted context omitted.

I really like that example, but I am wondering if it would really be true? Real drivers would not maintain a perfect speed, but would instead work to maintain the average. If you looked closely at the speed, it would drift away from the average, then the peddle would move to return it to the average. So it would look a bit like an integral (The I in PID control) of the difference from the mean speed right?

Well, we're obviously talking about perfectly spherical drivers (good point though).

Yes I figured :) Just pointing out that reality is always a bit more nuanced. To put it more simply, the petal position could be seen as an error accumulator.

Re: Everything Is Correlated

#40

Post by yeahitslikethat »

ajuc wrote: post #7

When the correlation is close to 0 it's often because of a feedback loop. For example - in economy with central bank trying to hit inflation target - interest rates and inflation will have near 0 correlation (interest rates change but inflation remains constant). That's because central bank adjusts interest rates to counter other variables so that inflation remains near the target. Other example (my favorite, it was…

Pressing the brake is positively correlated with the car going faster. Down hill.

Good thing correlation is not an indicator of causation.

Post reply on HN