Programming competitions correlate negatively with being good on the job • Hacker News Board

Re: Programming competitions correlate negatively with being good on the job

#101

Post by fapjacks » Mon, Apr 06, 2015, 12:05 AM UTC

oskarth wrote: post #75

sytelus wrote: post #72

Earlier quoted context omitted.

Being a champion at something requires excruciating narrow focus on something for unusually long time. If you are getting GPA of 4.0 or play Rachmaninoff's Piano Concerto No 3 or deadlift 400 pounds or in top 1000 chess players - you probably have to work on it for hours a day for years while ignoring everything else (of course unless you are one of those 1 in million polymath). People getting in to final rounds of p…

In general these things tend to follow a logistic curve. It's a useful meta-skill to know when investing more time has marginal utility. I think your examples are great, if you just change deadlift 400 pounds to maybe ~500-600 pound, (or roughly 2.5-3x bodyweight for adult men) [1]. Most adult men can reach a 400 pound deadlift in less than 2-3 years of lifting (that's I mention this only because a lot of people here…

You know, interestingly and according to my own personal anecdote, I feel significantly more efficient and productive in programming after I started weightlifting every day and using a standing desk at work. I'd guess between 15% and 25% more effective as a programmer. What I can figure is that I began to challenge more of the world around me (by lifting every day and running 10k or more on the weekend), and the same determination I use to motivate this activity has "bled over" into my programming. This is all 100% anecdote and purely unscientific, but it's how I feel after at least a couple of on-again-off-again exercise periods in my life, culminating finally in what seems to be a more permanent pattern of physical fitness. I really believe all humans should engage in weightlifting (and not simply a bit of cardio).

Re: Programming competitions correlate negatively with being good on the job

#102

Post by mikepavone » Mon, Apr 06, 2015, 12:10 AM UTC

sytelus wrote: post #72

oskarth wrote: post #23

It's still a positive predictor in general. It just means that, beyond a certain point, it's a negative predictor. Just like a GPA of 3.5+ is probably positively correlated with getting tenure, but maybe getting 4.0 is worse than getting 3.9. Watch the actual video to get the real take away: I think what it meant was that, everybody who gets hired at Google is pretty good. If I just had to pick someone off the street…

Being a champion at something requires excruciating narrow focus on something for unusually long time. If you are getting GPA of 4.0 or play Rachmaninoff's Piano Concerto No 3 or deadlift 400 pounds or in top 1000 chess players - you probably have to work on it for hours a day for years while ignoring everything else (of course unless you are one of those 1 in million polymath). People getting in to final rounds of p…

My personal experience with the ICFP programming competition suggests that you can do quite well with a fairly limited amount of time dedicated to it. My 2-man team managed to get 2nd place in the lightning round and 8th in the main round last year, despite using my side project programming language with its half-baked implementation. We generally do not do much preparation outside of having participated in the contest for a number of years. I did a bit of library work and my partner did a bit of a refresher on the language since he doesn't use it outside of the contest, but that was about it. Prior years were similar.

Granted, ICFP is a fair bit different than something like Google Code Jam. There are far fewer entries (typically around 200 or so for the main round), the task is larger, you have more time (3 days for the main round compared to 2.5 hours for a round of Code Jam) and you can compete as a team rather than an individual. I haven't really tried Code Jam yet so I can't say how much of that makes a difference.

Re: Programming competitions correlate negatively with being good on the job

#103

Post by skj » Mon, Apr 06, 2015, 12:12 AM UTC

danieltillett wrote: post #61

tedsanders wrote: post #42

Not to be gratuitously negative, but... All of these claims from Google that say competition performance hurts or that GPA doesn't matter are missing one huge thing: selection bias. Google only sees the performance of the employees that it hires, not the performance of the employees that it doesn't hire. Because of this, the data they analyze is statistically biased: all data is conditioned on being employed by Googl…

Great post. You are exactly right - all this study uncovers is that Google put too much bias in favour of programming competition wins in their hiring process in the past. What would be nice to know is exactly how important this factor was.

As I mentioned in another post, I don't think it's that Google sees programming competition success and says "oh boy, we gotta get this one!", but that the experience of those competitions helps the candidates perform better in interviews.

It is very hard to account for that in interviewing. Probably the only effective method would be to ding people with competition experience. Of course, that would be obvious conscious bias and would never happen.

Re: Programming competitions correlate negatively with being good on the job

#104

Post by diminoten » Mon, Apr 06, 2015, 12:13 AM UTC

jschwartzi wrote: post #93

diminoten wrote: post #87

Earlier quoted context omitted.

Seriously? Less than half of a software engineer's daily work is actually coding.

I mean what other components of your competence do the timed coding exercises reveal?

Prototyping, for one. Thought process. Creative thinking. Knowledge of relevant material.

It may not do these better than other tests, but basically any "solve this in x minutes" questions are going to reveal behavior in a person.

Re: Programming competitions correlate negatively with being good on the job

#105

Post by diminoten » Mon, Apr 06, 2015, 12:45 AM UTC

stretchwithme wrote: post #100

gwern wrote: post #33

Earlier quoted context omitted.

Yeah, this might be an instance of http://en.wikipedia.org/wiki/Berkson%27s_paradox - correlations after selection/conditioning may be different than before. Not well known, but relevant to a lot of the stuff about Google's hiring in particular (eg I wasted a lot of time explaining to people 2 years ago or so that Google's HR guy saying that intelligence didn't matter to their hiring decisions != intelligence doesn't…

Intelligence SHOULD have mattered when that HR guy was hired. Perhaps he means it in the Carol Dweck sense, that how we view our failures is what determines our success. IQ tests only measure what we've learned already, not our attitude towards learning.

No, you're missing the whole point. Intelligence doesn't matter because everyone is smart who applies to Google.

Re: Programming competitions correlate negatively with being good on the job

#106

Post by o_syn » Mon, Apr 06, 2015, 8:07 AM UTC

sytelus wrote: post #72

oskarth wrote: post #23

It's still a positive predictor in general. It just means that, beyond a certain point, it's a negative predictor. Just like a GPA of 3.5+ is probably positively correlated with getting tenure, but maybe getting 4.0 is worse than getting 3.9. Watch the actual video to get the real take away: I think what it meant was that, everybody who gets hired at Google is pretty good. If I just had to pick someone off the street…

Being a champion at something requires excruciating narrow focus on something for unusually long time. If you are getting GPA of 4.0 or play Rachmaninoff's Piano Concerto No 3 or deadlift 400 pounds or in top 1000 chess players - you probably have to work on it for hours a day for years while ignoring everything else (of course unless you are one of those 1 in million polymath). People getting in to final rounds of p…

""" Q: What do you hate about coding?

A: "Software engineering". Most real-world programming doesn't involve any new algorithm, doesn't require any cleverness; it merely involves implementing some known solution and making sure it's robustly commented and tested. It requires self-discipline simply because it's so boring! """ [1]

SnapDragon, who's current rating is 3005 on Topcoder, and was ranked 18 on that site said that, who by the way is employed at Google (at least he was when he gave that interview).

I think that pretty much sums it up. In addition, there is an aspect of software engineering that is much like playing with soap bubbles. All software eventually becomes outdated and unusable. If you're very lucky, some of the code you wrote will be used in future versions and maybe last for a decade or two before even that is completely discarded and forgotten.

So it is completely unclear that anyone would want to write production code for the entertainment value. And secondly, most real world programming is in fact pretty boring, and it is likely that solving puzzles is far more entertaining.

To draw an analogy, some smart people don't do well in school because you are graded on things like penmanship and mentioning the units and things like that. I don't think the scenario here is vastly different.

[1] http://community.topcoder.com/tc?module=Static&d1=features&d...

Re: Programming competitions correlate negatively with being good on the job

#107

Post by ScottBurson » Mon, Apr 06, 2015, 7:40 PM UTC

Interesting. My only anecdatum runs directly counter to Norvig's hypothesis. Some years ago, in my last job, we hired into my group a former TopCoder winner. He turned out to be the most careful, patient programmer I have ever met. It would seem to take him a long time to get things done, and sometimes I got a little frustrated with this. But he worked for us for about two years, and wrote a significant though not massive amount of code; and to my knowledge, he never shipped a bug. Not one.

I think tedsanders has the right answer.

Re: Programming competitions correlate negatively with being good on the job

#108

Post by collyw » Mon, Apr 06, 2015, 8:01 PM UTC

diminoten wrote: post #104

jschwartzi wrote: post #93

Earlier quoted context omitted.

I mean what other components of your competence do the timed coding exercises reveal?

Prototyping, for one. Thought process. Creative thinking. Knowledge of relevant material. It may not do these better than other tests, but basically any "solve this in x minutes" questions are going to reveal behavior in a person.

Why does crate thinking have to done fast? I find my best and most creative solutions come randomly after thinking about them "in the background" for a while.

Re: Programming competitions correlate negatively with being good on the job

#109

Post by PetrSavelev » Tue, Apr 07, 2015, 4:41 AM UTC

ScottBurson wrote: post #107

Interesting. My only anecdatum runs directly counter to Norvig's hypothesis. Some years ago, in my last job, we hired into my group a former TopCoder winner. He turned out to be the most careful, patient programmer I have ever met. It would seem to take him a long time to get things done, and sometimes I got a little frustrated with this. But he worked for us for about two years, and wrote a significant though not ma…

If you are not careful while solving problems in programming contests you are screwed. Because if you have limited time you don't have time too find problems.

Re: Programming competitions correlate negatively with being good on the job

#110

Post by danieltillett » Tue, Apr 07, 2015, 5:37 AM UTC

skj wrote: post #103

danieltillett wrote: post #61

Earlier quoted context omitted.

Great post. You are exactly right - all this study uncovers is that Google put too much bias in favour of programming competition wins in their hiring process in the past. What would be nice to know is exactly how important this factor was.

As I mentioned in another post, I don't think it's that Google sees programming competition success and says "oh boy, we gotta get this one!", but that the experience of those competitions helps the candidates perform better in interviews. It is very hard to account for that in interviewing. Probably the only effective method would be to ding people with competition experience. Of course, that would be obvious consci…

I don't think programming competition wins (PWG) are a negative when selecting candidates out of the potential pool, but Google must have been over-weighting the value of PWG when selecting their candidates. Basically the candidates they hired with PWG must have been weaker overall than the other people they hired. To put it another way PWG might explain 5% of the variations between hires, but if it was given a weighting of 20% then we would see the negative correlation observed.