Three hundred programming interviews in thirty days • Hacker News Board

Re: Three hundred programming interviews in thirty days

#11

Post by majormajor » Tue, Jun 23, 2015, 7:24 PM UTC

It seems like they're only evaluating phone screen methods against their pre-designed coding interview problem? But what if there are issues with that problem?

There seems to be a big assumption that "our programming questions are going to be good and predictive, even if everyone else's are bad." What if being able to describe in-depth a past (real) project correlates just as well (or better) to on-the-job performance as being able to design and code one of their artificial ones? Or what if those artificial ones just don't correlate that well with on-the-job performance in the first place?

It is definitely harder to BS-detect/grade, though.

They want to re-run against actual job performance in the future, that's nice, but it seems like they're throwing ideas out awfully early, then.

Re: Three hundred programming interviews in thirty days

#12

Post by notacoward » Tue, Jun 23, 2015, 7:25 PM UTC

Here's the part that really seems to matter the most.

    This does create some danger of circular reasoning
    (perhaps we're just carefully describing our own biases).
    But we have to start somewhere, and basing our 
    evaluations on how people write actual code seems like a
    good place.The really exciting point comes when we can 
    re-run all this analysis, basing it on actual job
    performance, rather than interview results.

Absolutely. Results on the earlier screens and results on the later interview aren't exactly independent variables, and neither is the one that really seems to matter - subsequent on-the-job success. There are all sorts of biases and confounding factors likely to be shared between them, especially since there's no indication that the later interviews were even done blind w.r.t. the earlier screens. Until then, we're just measuring correlations between different interview techniques, and it should be no surprise that two different kinds of code-focused interviews show the highest correlation.

Re: Three hundred programming interviews in thirty days

#13

Post by donall » Tue, Jun 23, 2015, 7:25 PM UTC

michaelvkpdx wrote: post #5

Without any demographic data about who the interviewees were, this is basically useless. Are they all under 30, white and Asian men? Where?

I can see that that would be interesting information to have, but why is it absolutely necessary?

Re: Three hundred programming interviews in thirty days

#14

Post by evanpw » Tue, Jun 23, 2015, 7:25 PM UTC

jameshart wrote: post #3

Correlation between testing and interviewing success tells you nothing about the correlation between either of them and actual work performance.

Even so, an automated quiz that could predict with high confidence that someone would fail an interview would be a big win, saving both interviewers and interviewees a lot of time.

Re: Three hundred programming interviews in thirty days

#15

Post by ammon » Tue, Jun 23, 2015, 7:25 PM UTC

jasode wrote: post #4

Imo, this post did not organize its data and findings into a coherent presentation. For example... >The fizzbuzz-style coding problems, however, did not perform as well. While the confidence intervals are large, the current data shows less correlation with interview results. [...] The coding problems were also harder for people to finish. We saw twice the drop off rate on the coding problems as we saw on the quiz. I…

Hey! Author here. Yes, we did not know that the screening steps were meaningful, so for the first 300 applicants, we interviewed everyone, even people who performed badly. We then looked for correlation between screening step scores, and programming interview results. Doing well on the fizzbuzz problems was not very correlated.

By dropoff I mean people who left during a step, and never logged back into our site.

Re: Three hundred programming interviews in thirty days

#16

Post by eitally » Tue, Jun 23, 2015, 7:26 PM UTC

protonfish wrote: post #6

It's great to see some objective research being done on this, and I am very interested in following the results. They mention evaluating the effectiveness of giving a candidate a project to do "in their own time." I recently had a interview that included this and I can share the result: I accepted an offer from a different company that didn't require it. I doubt my life is that different than anyone else's, with a fu…

My wife is in this position now, and it's not for a programming job. She works in pharmaceuticals, in pharmacovigilance & risk management. She does a lot of data analysis & writing (the latest report she wrote for Health Canada & the FDA was 461 pages).

A company she's interviewing with asked her to do a sample writing task as part of her application/interview. She hasn't decided whether it's worth it or not. For someone who has 15 years experience in the field, excellent recommendations and credentials, spending (her estimate) 12-15 hours doing pro bono work just to apply somewhere is ridiculous.

Re: Three hundred programming interviews in thirty days

#17

Post by choppaface » Tue, Jun 23, 2015, 7:27 PM UTC

I've done over 1000 interviews and my experience agrees with their findings that talking about a project is not a good predictor of coding. I usually left detailed resume questions for the end since so many candidates would bomb the coding part of the interview.

I'm surprised they didn't get stronger results from fizz buzz, but I noticed among the candidates I saw that the percentage of 'non-coders' is substantial but not a majority.

One thing missing from this investigation is a measure of solution quality. A good portion of candidates who actually finished coding questions with me ended without thoroughly understanding how their code worked and/or had code that would be hard to maintain. Other candidates would write top-notch code but were unable to explain their thought process to some extent. These are critical pieces to the interview that contribute much more 'color' than 'score' and are important to note.

Re: Three hundred programming interviews in thirty days

#18

Post by ammon » Tue, Jun 23, 2015, 7:28 PM UTC

notacoward wrote: post #12

Here's the part that really seems to matter the most. This does create some danger of circular reasoning (perhaps we're just carefully describing our own biases). But we have to start somewhere, and basing our evaluations on how people write actual code seems like a good place.The really exciting point comes when we can re-run all this analysis, basing it on actual job performance, rather than interview results. Abso…

Author here. We do do the interviews blind to earlier screening results. That's clearly vital. But you're totally right that job performance is the real thing that matters

Re: Three hundred programming interviews in thirty days

#19

Post by mbesto » Tue, Jun 23, 2015, 7:28 PM UTC

> In our first 30 days, we've come up with a replacement for resume screens, and shown that it works well.

What's the metric that shows it works well?

> The really exciting point comes when we can re-run all this analysis, basing it on actual job performance, rather than interview results.

And how precisely do you measure job performance? If this is achievable, I've got a line of companies out my door that would love to pay for a service that systematically measures job performance.

Re: Three hundred programming interviews in thirty days

#20

Post by notacoward » Tue, Jun 23, 2015, 7:29 PM UTC

ammon wrote: post #18

notacoward wrote: post #12

Here's the part that really seems to matter the most. This does create some danger of circular reasoning (perhaps we're just carefully describing our own biases). But we have to start somewhere, and basing our evaluations on how people write actual code seems like a good place.The really exciting point comes when we can re-run all this analysis, basing it on actual job performance, rather than interview results. Abso…

Author here. We do do the interviews blind to earlier screening results. That's clearly vital. But you're totally right that job performance is the real thing that matters

Cool. Might want to mention that in the article, since I'm unlikely to be the only person who thinks of that.