Programming competitions correlate negatively with being good on the job • Hacker News Board

Re: Programming competitions correlate negatively with being good on the job

#71

Post by Ruberik » Sun, Apr 05, 2015, 7:19 PM UTC

joe_the_user wrote: post #67

oskarth wrote: post #23

It's still a positive predictor in general. It just means that, beyond a certain point, it's a negative predictor. Just like a GPA of 3.5+ is probably positively correlated with getting tenure, but maybe getting 4.0 is worse than getting 3.9. Watch the actual video to get the real take away: I think what it meant was that, everybody who gets hired at Google is pretty good. If I just had to pick someone off the street…

I don't think that should take away from the argument of the article, however: programming contest winners are used to cranking solutions out fast and that you performed better at the job if you were more reflective and went slowly and made sure things were right. The idea that a programmer can be very good at one level and have that goodness fail at other levels is important to consider. It is important to consider…

I think it's fair to say that programming contest winners are exceptionally good at cranking solutions out fast; but they could be just as good at working reflectively in a team as everyone else, or even better.

The author of this article appears to think that he or she has provided evidence for the sentence you quoted, but a number of comments in this thread have pointed out that isn't what the evidence says; so the quoted sentence is simply speculation.

Re: Programming competitions correlate negatively with being good on the job

#72

Post by sytelus » Sun, Apr 05, 2015, 7:22 PM UTC

oskarth wrote: post #23

It's still a positive predictor in general. It just means that, beyond a certain point, it's a negative predictor. Just like a GPA of 3.5+ is probably positively correlated with getting tenure, but maybe getting 4.0 is worse than getting 3.9. Watch the actual video to get the real take away: I think what it meant was that, everybody who gets hired at Google is pretty good. If I just had to pick someone off the street…

Being a champion at something requires excruciating narrow focus on something for unusually long time. If you are getting GPA of 4.0 or play Rachmaninoff's Piano Concerto No 3 or deadlift 400 pounds or in top 1000 chess players - you probably have to work on it for hours a day for years while ignoring everything else (of course unless you are one of those 1 in million polymath).

People getting in to final rounds of programming contests are usually not the best programmers in the world in terms of breadth, maturity and exposure to other aspects of programming. They typically have put enormous effort on super optimizing their mental structure to do nothing but learn to master every little trick necessary to win the contest. This typically means memorizing implementation of pretty much all popular solutions in these contests, mastering odd tricks while ignoring general insights, less attention on elegance and maintainability, less incentivized to work in group, more tuned to getting any solution as quickly as possible instead of striving to get best possible solution, ignoring any areas that are not popular in such contests etc. This kind of brain training obviously has its negative. However on average, winners from contests will usually out perform your average programmer any day, they just won't outperform our super stars who would probably disqualify well before semi-finals because they were busy in learning other things that are not useful at competitions.

My observations from dealing with one such person (I know, a great statistical sample size) very closely was that they are extremely fast learners and usually gravitate toward working on small very well defined problem that they already know how to solve but they suck at getting big picture, learning something large and complex and understanding the intricate interdependencies involved in complex systems.

Re: Programming competitions correlate negatively with being good on the job

#73

Post by sukilot » Sun, Apr 05, 2015, 7:40 PM UTC

benhamner wrote: post #68

This is flat out wrong. It disagrees with every other data point I have, so I'm very skeptical with both the methodology (which is opaque) and the conclusion. From all of my experience at Kaggle (we run machine learning competitions), with our community, and from being close to programming competition sites & understanding their communities, doing great at competitions is an unambiguously positive signal. (It's worth…

Kaggle is new and runs deeper contests than TopCoder and CodeJam of the 2000s.

Re: Programming competitions correlate negatively with being good on the job

#74

Post by sukilot » Sun, Apr 05, 2015, 7:43 PM UTC

namelezz wrote: post #34

Google keeps proving itself as a king of hyping. Their employees periodically make sound statement regarding to winning programming competitions as not being a good indication of effective employees but they only consider hiring those who can crack programming competition questions during their short interview session. Good for you if you want to judge a fish by its ability to climb a tree.

Google interviews are quite different from programming contests. In a programming contest, there is little time to think or talk, success is about instantly pattern matching problem to solution , copy pasting library Code, and pounding out a solution.

Interviews are 30min of discussion and 20 lines of code.

Re: Programming competitions correlate negatively with being good on the job

#75

Post by oskarth » Sun, Apr 05, 2015, 7:50 PM UTC

sytelus wrote: post #72

oskarth wrote: post #23

It's still a positive predictor in general. It just means that, beyond a certain point, it's a negative predictor. Just like a GPA of 3.5+ is probably positively correlated with getting tenure, but maybe getting 4.0 is worse than getting 3.9. Watch the actual video to get the real take away: I think what it meant was that, everybody who gets hired at Google is pretty good. If I just had to pick someone off the street…

Being a champion at something requires excruciating narrow focus on something for unusually long time. If you are getting GPA of 4.0 or play Rachmaninoff's Piano Concerto No 3 or deadlift 400 pounds or in top 1000 chess players - you probably have to work on it for hours a day for years while ignoring everything else (of course unless you are one of those 1 in million polymath). People getting in to final rounds of p…

In general these things tend to follow a logistic curve. It's a useful meta-skill to know when investing more time has marginal utility.

I think your examples are great, if you just change deadlift 400 pounds to maybe ~500-600 pound, (or roughly 2.5-3x bodyweight for adult men) [1]. Most adult men can reach a 400 pound deadlift in less than 2-3 years of lifting (that's I mention this only because a lot of people here on HN are probably lacking physical activity, compared to what human were evolutionary selected for, and it's useful to know what can be expected. It's generally more than most people think.

1: See for example http://chrissalvato.com/2009/12/skill-guidelines-for-buildin...

Re: Programming competitions correlate negatively with being good on the job

#76

Post by walshemj » Sun, Apr 05, 2015, 8:02 PM UTC

sytelus wrote: post #72

oskarth wrote: post #23

It's still a positive predictor in general. It just means that, beyond a certain point, it's a negative predictor. Just like a GPA of 3.5+ is probably positively correlated with getting tenure, but maybe getting 4.0 is worse than getting 3.9. Watch the actual video to get the real take away: I think what it meant was that, everybody who gets hired at Google is pretty good. If I just had to pick someone off the street…

Being a champion at something requires excruciating narrow focus on something for unusually long time. If you are getting GPA of 4.0 or play Rachmaninoff's Piano Concerto No 3 or deadlift 400 pounds or in top 1000 chess players - you probably have to work on it for hours a day for years while ignoring everything else (of course unless you are one of those 1 in million polymath). People getting in to final rounds of p…

Yes the real word tends to value thing like your phone bill being absolutely correct and auditable and not prone to bizarre edge cases that the completion winners don't care about

Re: Programming competitions correlate negatively with being good on the job

#77

Post by hyperpape » Sun, Apr 05, 2015, 8:20 PM UTC

Ruberik wrote: post #17

The conclusion this article offers is drawn from a badly flawed study from over a decade ago, which Google decided neither to publish nor to use the conclusions from. A machine learning system was trained at Google as an attempt to do "scientific hiring." As their ground truth, the team used the performance reviews of people who had already been hired at Google . To optimistically give it the benefit of every doubt,…

"To optimistically give it the benefit of every doubt, this study says that Google put too much emphasis on success in programming competitions when it was making hiring decisions in the early 2000s."

It's kind of unbelievable that anyone is even bothering to say _anything_ in this thread without addressing this point. Of course people sometimes overvalue competitions, and if it's true that Google did it, that would explain the effect whether competition winners are typically better or worse than other people who interview at Google.

Do I know it's true? Of course not. But it's such an obvious question to ask that your first question should be "how did measure/rule out that effect?". And if there's no answer, you go back to the drawing board and try to answer it.

Re: Programming competitions correlate negatively with being good on the job

#78

Post by hmate9 » Sun, Apr 05, 2015, 8:25 PM UTC

yodsanklai wrote: post #19

I thought that Google interview process was similar to this type of programming competitions.

Haha, exactly! That's what I find strange too.

Re: Programming competitions correlate negatively with being good on the job

#79

Post by Gibbon1 » Sun, Apr 05, 2015, 8:26 PM UTC

joe_the_user wrote: post #67

oskarth wrote: post #23

It's still a positive predictor in general. It just means that, beyond a certain point, it's a negative predictor. Just like a GPA of 3.5+ is probably positively correlated with getting tenure, but maybe getting 4.0 is worse than getting 3.9. Watch the actual video to get the real take away: I think what it meant was that, everybody who gets hired at Google is pretty good. If I just had to pick someone off the street…

I don't think that should take away from the argument of the article, however: programming contest winners are used to cranking solutions out fast and that you performed better at the job if you were more reflective and went slowly and made sure things were right. The idea that a programmer can be very good at one level and have that goodness fail at other levels is important to consider. It is important to consider…

One of the things I noticed in school and later working is a lot of people excel when the problem is spoon fed to them. When the constraints and and desired answer are very mechanical. As it gets fuzzier, a lot of people even 'smart' people start having real issues. And you see the usual cognitive blockers kick in. 'frustration, fear, anger'

Meaning, the usual academic problem where you are given the inputs, and only the inputs you need, and a desired solution, doesn't actually match up that well with real world problems.

The other thing I've seen is a _lot_ of formally trained engineers are motivated by puzzles. Once they've solved it, all the other things that apply to the real world, like deadlines, shipping projects, making customers happy, and making money for the company are not the least amount interesting to them.

Re: Programming competitions correlate negatively with being good on the job

#80

Post by [deleted] » Sun, Apr 05, 2015, 8:36 PM UTC

oskarth wrote: post #75

sytelus wrote: post #72

Earlier quoted context omitted.

Being a champion at something requires excruciating narrow focus on something for unusually long time. If you are getting GPA of 4.0 or play Rachmaninoff's Piano Concerto No 3 or deadlift 400 pounds or in top 1000 chess players - you probably have to work on it for hours a day for years while ignoring everything else (of course unless you are one of those 1 in million polymath). People getting in to final rounds of p…

In general these things tend to follow a logistic curve. It's a useful meta-skill to know when investing more time has marginal utility. I think your examples are great, if you just change deadlift 400 pounds to maybe ~500-600 pound, (or roughly 2.5-3x bodyweight for adult men) [1]. Most adult men can reach a 400 pound deadlift in less than 2-3 years of lifting (that's I mention this only because a lot of people here…

[deleted]