Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Programming competitions correlate negatively with being good on the job

catonmat.net

Post reply on HN
Print view View original

61–70 of 112 posts

Re: Programming competitions correlate negatively with being good on the job

#61

Post by danieltillett »

tedsanders wrote: post #42

Not to be gratuitously negative, but... All of these claims from Google that say competition performance hurts or that GPA doesn't matter are missing one huge thing: selection bias. Google only sees the performance of the employees that it hires, not the performance of the employees that it doesn't hire. Because of this, the data they analyze is statistically biased: all data is conditioned on being employed by Googl…

Great post. You are exactly right - all this study uncovers is that Google put too much bias in favour of programming competition wins in their hiring process in the past. What would be nice to know is exactly how important this factor was.

Re: Programming competitions correlate negatively with being good on the job

#62

Post by diminoten »

collyw wrote: post #46
diminoten wrote: post #39

Earlier quoted context omitted.

Why must it be "how you work on a day to day basis"? Why must interviews only be about exactly what you're being hired to do?

I would like to think it is at least somewhat related to what I am doing, rather than a lucky dip of a question that I might get or might not. I am a way better software engineer than 15 years ago, yet I am worse at these sort of "challenges".

There's a lot more to being a software engineer than coding competence, and timed questions reveal the non-coding competence parts of your abilities more than anything else.

Perhaps you're not as good as you think.

Re: Programming competitions correlate negatively with being good on the job

#63

Post by nicolasp »

JacobEdelman wrote: post #55

I'd be interested to know what type of programming competition is being talked about here. There is a world of difference between Hackathons and Algorithmic style competitions.

They're talking about the algorithmic competitions like ACM ICPC or TopCoder. Google have their own (Google Code Jam), which they use as a recruiting tool.

Re: Programming competitions correlate negatively with being good on the job

#64

Post by codingdave »

svisser wrote: post #4

Coming up with clever solutions is indeed very different from coming up with maintainable solutions.

... and many of the programmers I find in the corporate world do neither. But that is probably for another discussion.

Re: Programming competitions correlate negatively with being good on the job

#65

Post by venomsnake »

codingdave wrote: post #64
svisser wrote: post #4

Coming up with clever solutions is indeed very different from coming up with maintainable solutions.

... and many of the programmers I find in the corporate world do neither. But that is probably for another discussion.

Why code when you can go full dilbert on corporate politics.

Re: Programming competitions correlate negatively with being good on the job

#66

Post by kentonv »

orborde wrote: post #52
Ruberik wrote: post #17

The conclusion this article offers is drawn from a badly flawed study from over a decade ago, which Google decided neither to publish nor to use the conclusions from. A machine learning system was trained at Google as an attempt to do "scientific hiring." As their ground truth, the team used the performance reviews of people who had already been hired at Google . To optimistically give it the benefit of every doubt,…

Citation?

Ruberik ran Google Code Jam for several years. (Or maybe someone registered his handle on HN just to forge this comment, but that seems like a silly thing to do.)

EDIT: To be clear, Ruberik is NOT an impostor, and it's sad that someone is now going around flagging his comments.

Re: Programming competitions correlate negatively with being good on the job

#67

Post by joe_the_user »

oskarth wrote: post #23

It's still a positive predictor in general. It just means that, beyond a certain point, it's a negative predictor. Just like a GPA of 3.5+ is probably positively correlated with getting tenure, but maybe getting 4.0 is worse than getting 3.9. Watch the actual video to get the real take away: I think what it meant was that, everybody who gets hired at Google is pretty good. If I just had to pick someone off the street…

I don't think that should take away from the argument of the article, however:

programming contest winners are used to cranking solutions out fast and that you performed better at the job if you were more reflective and went slowly and made sure things were right.

The idea that a programmer can be very good at one level and have that goodness fail at other levels is important to consider. It is important to consider that a combination of skills, from cranking code to considering the impact of that code, matter. This consideration can lead to an entire team being valued rather than, say, the glorification of the "rock star" programmer.

Re: Programming competitions correlate negatively with being good on the job

#68

Post by benhamner »

This is flat out wrong.

It disagrees with every other data point I have, so I'm very skeptical with both the methodology (which is opaque) and the conclusion.

From all of my experience at Kaggle (we run machine learning competitions), with our community, and from being close to programming competition sites & understanding their communities, doing great at competitions is an unambiguously positive signal.

(It's worth noting that doing great at competitions is only a positive signal - the lack of competitions is by no means a negative signal).

Many of our customers have found that their best hires have come from competitions. In a lot of cases, this surfaces candidates that would normally be completely overlooked because they don't fit the "top tier CS school" mold that recruiters commonly overfit to.

Several companies have had a successful recruiting strategy built on poaching our top users (https://www.kaggle.com/users).

Peter Norvig's criticism that "programming contest winners are used to cranking solutions out fast and that you performed better at the job if you were more reflective and went slowly and made sure things were right" is specific to programming competitions with very short time durations (vs. the machine learning competitions that I'm used to running, which typically last months and incentivize solutions that generalize well).

However, we've seen that many programming competition winners also do well on machine learning competitions, and the same qualities that aid in competitive programming (creativity, efficiency, tenacity, fluidity with tools, and the ability to build something that works) help win machine learning competitions.

Re: Programming competitions correlate negatively with being good on the job

#69

Post by asdfologist »

benhamner wrote: post #68

This is flat out wrong. It disagrees with every other data point I have, so I'm very skeptical with both the methodology (which is opaque) and the conclusion. From all of my experience at Kaggle (we run machine learning competitions), with our community, and from being close to programming competition sites & understanding their communities, doing great at competitions is an unambiguously positive signal. (It's worth…

I think you're conflating positive correlation with getting hired vs positive correlation with doing well on the job (once hired). See oskarth's comment at the top. No one disputes that doing great at competitions is a positive predictor for getting hired.

Re: Programming competitions correlate negatively with being good on the job

#70

Post by Ruberik »

orborde wrote: post #52
Ruberik wrote: post #17

The conclusion this article offers is drawn from a badly flawed study from over a decade ago, which Google decided neither to publish nor to use the conclusions from. A machine learning system was trained at Google as an attempt to do "scientific hiring." As their ground truth, the team used the performance reviews of people who had already been hired at Google . To optimistically give it the benefit of every doubt,…

Citation?

I don't have any evidence of this other than my word, sorry. As kentonv points out, I was in a position to know: in the early days of Code Jam, we to do some convincing of Googlers who had seen the "programming contestants don't perform well" headline.

Also, as others have pointed out, Peter Norvig mentions some of this in his talk.

Post reply on HN