Programming competitions correlate negatively with being good on the job

Not to be gratuitously negative, but...

All of these claims from Google that say competition performance hurts or that GPA doesn't matter are missing one huge thing: selection bias.

Google only sees the performance of the employees that it hires, not the performance of the employees that it doesn't hire. Because of this, the data they analyze is statistically biased: all data is conditioned on being employed by Google. So when Google says things like "GPA is not correlated with job performance" what you should hear is "Given that you were hired by Google, GPA is not correlated with job performance."

In general, when you have some thresholding selection, it will cause artificial negative correlations to show up. Here's a very simple example that I hope illustrates the point: Imagine a world where high school students take only two classes, English and Math, and they receive one of two grades, A or B. Now imagine a college that admits students with at least one A (AB, BA, or AA) and that rejects everyone without an A (BB). Now imagine that there is absolutely zero correlation between Math and English - performance on one is totally independent of the other. However, when the college looks at their data, they will nonetheless see a stark anticorrelation between Math and English grades (because everyone who has a B in one subject always has an A in the other subject, simply because all the BBs are missing from their dataset).

The bottom line is that whenever you have some score that is some positive combination of input variables and then you threshold your observed data on a minimum total score (as is the case in hiring at Google or in college admissions), then you will see much stronger negative correlations between your inputs than exists in real life.

And really, whenever you run some selection algorithm, you should hope that (on the margin) there are no correlations between your selection decision and your inputs. If there still is a correlation post-selection, that means your algorithm has left money on the table. So when Google says that programming competitions are negatively correlated with performance and GPA is uncorrelated with performance, what that likely means is that Google's hiring overvalues programming competitions and fairly values GPA.

In fact, if we did a randomized controlled study (the gold standard of proving causation), I think we'd see the expected results. Just imagine - if you grabbed two random programmers from the entire population, one who had won a competition and one who had not, do you really think the competition winner would be the inferior programmer?

Edit: Many other posts here are coming up with plausible sounding narratives to fit the data. "Competition winners are one-trick ponies or loners or write awful code." I encourage everyone to think critically about the data analysis itself.

Edit2: From Gwern's comment, this phenomenon is apparently called Berkson's paradox: http://en.wikipedia.org/wiki/Berkson%27s_paradox

Driven by what? If it is driven by title then you have something like this http://www.nextbigwhat.com/indian-developers-accused-of-chea...

I would like to think it is at least somewhat related to what I am doing, rather than a lucky dip of a question that I might get or might not. I am a way better software engineer than 15 years ago, yet I am worse at these sort of "challenges".

What none of these tests, contest or not, can filter for is long term work ethic, dedication and creativity. Of course, these may not be important for all jobs. It is easy for some personality tyles to get excited and focused in the context of an event. It is a very different story to dedicate yourself intensely to a project or a codebase for years. I know people who suck at tests who blow away just about anyone due to the intense dedication to the job they can deliver on a consistent basis.

A different angle - from someone averse to competitions and trading of most kinds: both have simple cost functions, unlike most of the rest of life. Yeah, it will tend to simplify politics which I think is a good thing. But in other areas a large part of one's effort goes into figuring out what to do - what the cost function actually is, into expanding the understanding of what it is.

Incidentally, much of human intelligence and talent in many areas manifests in understanding what to do; it's what machine intelligence is worst at right now; and it's the thing without which machines often beat people (board games are a simple example.)

So there are downsides to "simple politics" or rather simple cost functions; I for one don't care about such problems very much.