[deleted]
Three hundred programming interviews in thirty days
131–140 of 248 posts
Re: Three hundred programming interviews in thirty days
#132"The really exciting point comes when we can re-run all this analysis, basing it on actual job performance, rather than interview results" I'm not sure how this gets around the circularity arguments though, since you never get to evaluate the job performance of someone you selected out already. Only the tiny fraction of coders that make it past the initial test get evaluated, which could serve to reinforce the potent…
Lack of insight into false negatives is (I think) the major flaw in most studies of hiring engineers. For the analysis in this blog post, we just passed everyone thorough the screening, to avoid the issue. Going forward, we'll be passing 10% of those who fail the screening, to be able to track our false negative rate. My dream is to reach a scale where can do the same thing with the final hiring decisions (well, less than 10%, clearly). Someone needs to do this.
Re: Three hundred programming interviews in thirty days
#133Earlier quoted context omitted.
I did this once. After a take home quiz that took about an hour, I had a 6-7 hour "homework" problem. I did it, and sent it in. It took the company a month to get back to me, though I did stay in contact with the recruiter. The final answer, delivered by the recruiter, was "we've decided not to more forward at this time." No other feedback. I have no idea if anyone even looked at it - I never did speak to a dev at th…
I've seen several companies giving out such homework to me or my friends back in college. And to be honest, none of the are among the "great culture company list" people are talking about. Beside all the issues you mentioned above, those tests don't have a clear PASS line. You spent hours on that and can be rejected for no reason. Back in college I've done a code challenge from Box and my solution ranked No.1 in thei…
Even in C I would point out that the compiler will likely make such a trivial optimization for you. That was clearly a company full of morons.
Re: Three hundred programming interviews in thirty days
#134Earlier quoted context omitted.
The point is that they pay you the $100k for the time you give them as an employee. They don't pay you a cent for the interview project, so you shouldn't give them a single second in return.
They don't pay you for the interview, but that usually costs the company several man-hours worth of work just for the time an applicant is on-site. Should applicants be reimbursing companies for failed interviews?
Maybe. I've always thought it would be interesting to make applicants pay an application fee. This would cut down on people "spraying and praying" with their application, and would lessen the workload for companies. It would also justify spending more time and effort in reading applications, since the company isn't just wasting resources reading bad applications.
Even under the current system, at least the waste is symmetric. I give up a few hours of my time, and the company gives up a few hours of its time. There's equity. The mutual work that the company and applicant do offset each other.
A take-home test model skews that balance in favor of the company. An applicant can spend 10 or more hours working the project. The company can run it through an automated testing suite and have a recruiter spend five minutes looking it over. The system is designed to waste more of the applicant's time and less of the company's.
Re: Three hundred programming interviews in thirty days
#135Earlier quoted context omitted.
I did this once. After a take home quiz that took about an hour, I had a 6-7 hour "homework" problem. I did it, and sent it in. It took the company a month to get back to me, though I did stay in contact with the recruiter. The final answer, delivered by the recruiter, was "we've decided not to more forward at this time." No other feedback. I have no idea if anyone even looked at it - I never did speak to a dev at th…
I've seen several companies giving out such homework to me or my friends back in college. And to be honest, none of the are among the "great culture company list" people are talking about. Beside all the issues you mentioned above, those tests don't have a clear PASS line. You spent hours on that and can be rejected for no reason. Back in college I've done a code challenge from Box and my solution ranked No.1 in thei…
Not that does anything except to confirm the pointless of that selection criterion:
% python -mtimeit -s 'x=1' 'xRe: Three hundred programming interviews in thirty days
#136while this process feels like it's hitting the sweet spot for finding out who can write brilliant code, that's half or less of the battle in hiring people. personally (and as a hiring manager), i feel like a majority of the hiring process is dependent (obviously) on the environment you're hiring into.
hiring for that small startup? you'll want multi-hat wearing people first, brilliant programmers second.
hiring for a large enterprise team? you'll want to hire for "plays well with others" first, and brilliant programmers second.
that's not to say you should hire schleps, for sure. they should at least be competent programmers. i guess what i'm saying is (despite how it sounds), hiring someone who can program brilliantly is important, but not as important as hiring someone who can navigate your company's software-making requirements successfully.
firing the brilliant engineer who thinks he's more talented than everyone else in the small company so he keeps demanding to be put in charge? yup, that's a thing. firing the brilliant engineer who fights tooth-and-nail over some inconsequential feature the product team wants to change? that's a thing too. assigning a brilliant engineer to crap, meaningless work because no one else on the team wants to work with them? yuppers -- seen it.
in any organization, you are either the only one in charge or you're following someone else's orders -- both of which require different aspects around working well with others.
Re: Three hundred programming interviews in thirty days
#137It's great to see some objective research being done on this, and I am very interested in following the results. They mention evaluating the effectiveness of giving a candidate a project to do "in their own time." I recently had a interview that included this and I can share the result: I accepted an offer from a different company that didn't require it. I doubt my life is that different than anyone else's, with a fu…
I did this once. After a take home quiz that took about an hour, I had a 6-7 hour "homework" problem. I did it, and sent it in. It took the company a month to get back to me, though I did stay in contact with the recruiter. The final answer, delivered by the recruiter, was "we've decided not to more forward at this time." No other feedback. I have no idea if anyone even looked at it - I never did speak to a dev at th…
One take-home project I was given was very cleverly designed. Without going into too much detail, the project description initially seemed pretty simple. The actual getting-it-to-work part took about half an hour. However there was a requirement near the end that said the whole operation had to complete in less than X amount of time. The initial straightforward implementation I wrote took about 8-times longer to complete. Eventually, 3 hours later, after pulling out a lot of clever tricks I had come up with an implementation that met the performance requirements. The benefit of this type of approach is that it allows potential applicants of many levels to understand and tackle the problem while also allowing them to choose to bail out after that 30 minute mark if they can't think of any way to improve performance. Seems like a polite way to set a high bar standard.
Re: Three hundred programming interviews in thirty days
#138It's great to see some objective research being done on this, and I am very interested in following the results. They mention evaluating the effectiveness of giving a candidate a project to do "in their own time." I recently had a interview that included this and I can share the result: I accepted an offer from a different company that didn't require it. I doubt my life is that different than anyone else's, with a fu…
I did this once. After a take home quiz that took about an hour, I had a 6-7 hour "homework" problem. I did it, and sent it in. It took the company a month to get back to me, though I did stay in contact with the recruiter. The final answer, delivered by the recruiter, was "we've decided not to more forward at this time." No other feedback. I have no idea if anyone even looked at it - I never did speak to a dev at th…
> I had a 6-7 hour "homework" problem
Personally I would given them a link to my github which has hundreds of hours of work put into projects. From my experience a homework problem that long doesn't really prove anything (I could have outsourced it) and you never get feedback (this isn't the employer's fault but of laws in the US[1]).
I think a take home quiz is the only appropriate quiz if it's a basic less than an hour quiz. Harder than fizzbuzz but easier than creating a blog from scratch.
Don't even get me started on IQ tests or personality tests.
[1] - http://humanresources.about.com/od/selectemployees/qt/must-e...
Re: Three hundred programming interviews in thirty days
#139Earlier quoted context omitted.
I appreciate that you guys might not be statisticians, but if you're going to try and analyze data like this, you simply must address survival bias. As it stands, these data are meaningless unless you assume dropouts are completely unrelated to your screening. You claim doing well on the Fizzbuzz wasn't correlated with interview performance, but you also said "We saw twice the drop off rate on the coding problems as…
All candidates did both screens (quiz and fizzbuzz). The correlations were calculated against the same population. Now, I agree that survivor bias could affect the quality of these results (we know nothing about the significant % of people who dropped out). But it's not really possible to solve that problem outside of a lab. I don't think it's an argument to not do analysis. For now we're simply trying to minimize th…
I have to agree with kingmob. It very much sounds like survivor bias. My first reaction is that anyone who drops out due to a test has a high likelihood of dropping out because they can't do the test, which would leave you with a low correlation test when compared with the survivors, but a very high anti correlation with the total population.
I read a blog post a couple years ago by a game programmer/designer who outsources a lot of work through places like odesk/elance. Basically his thing was to weed out the fakers, he'd offer anyone ~5hrs at their bidding rate to finish a predefined programming task expected to take ~5hrs. He says this will usually drop his pool to less than 10 out of the hundreds who may apply, and he can usually use at least one of the people who complete the task. It's hard to say how many of these people go away because the task looks too big, and there's risk of not getting paid, but it's clearly a good filter for him.
As far as measuring this survivor bias, you might gain some insight by randomly altering the order of the testing. You could measure when people tended to drop off. You might even find that people all tend to drop off around the same amount of time, or maybe after some certain amount of effort. It might even be worth paying people to see if that would improve completion rates (while introducing it's own biases).
Re: Three hundred programming interviews in thirty days
#140Earlier quoted context omitted.
They don't pay you for the interview, but that usually costs the company several man-hours worth of work just for the time an applicant is on-site. Should applicants be reimbursing companies for failed interviews?
Maybe. I've always thought it would be interesting to make applicants pay an application fee. This would cut down on people "spraying and praying" with their application, and would lessen the workload for companies. It would also justify spending more time and effort in reading applications, since the company isn't just wasting resources reading bad applications. Even under the current system, at least the waste is s…
> Even under the current system, at least the waste is symmetric.
But it's never symmetric. Someone in HR spent time setting up the job posting and managing the process. Someone in management took the time to respond to HR and review your resum, then approve the interview. At least one person at a time is in the interview with you. Then the entire team will spend time afterward breaking it down.
A single hour spent by a bad candidate wastes at least 3 man-hours of work by the company, and most likely more.