If your company does coding tests during the engineering interview process, (how) do you measure the long term effectiveness of the tests? Do you keep internal metrics comparing candidates' score to their long term impact/success at the company? If yes, what have you learned from the results and how have those learnings impacted your hiring process?
Ask HN: Measuring the long-term benefit of interview code tests?
1–10 of 76 posts
Re: Ask HN: Measuring the long-term benefit of interview code tests?
#2and maybe a follow up question (to measure the false negatives)
Do you check the applicants who were denied based on their test and see where they ended up working at. E.g. you are a mid tier start up who rejects someone who ends up working at amazon as a high level engineer – do you mark that a failure?
Re: Ask HN: Measuring the long-term benefit of interview code tests?
#3For developers, coding tests that include deployment / infrastructure components (i.e., deploy your solution to a cloud container, or, build and compile your solution for desktop platform testing) are uniformly consistent with long-term impact / success. Problem solving at the algorithmic layer may be inversely correlated to success, if a candidate lacks a production skill set.
Unless one's focus is research and development, there is a non-zero cost to training for production skills, so it's best to start with someone who understands the delivery process.
Linear metrics are probably less useful, inasmuch as it will become rather obvious as to which employees are self-starting and work well with others, versus those that require motivation or are staunch individualists.
Re: Ask HN: Measuring the long-term benefit of interview code tests?
#4and maybe a follow up question (to measure the false negatives) Do you check the applicants who were denied based on their test and see where they ended up working at. E.g. you are a mid tier start up who rejects someone who ends up working at amazon as a high level engineer – do you mark that a failure?
I'd be careful to call that out as a negative; if the culture fit wasn't right, and the candidate would have been a net negative to the team, it shouldn't matter where they end up next, unless (of course) the candidate that was actually hired ends up being an even worse fit (ergo the need to fix your hiring process).
Re: Ask HN: Measuring the long-term benefit of interview code tests?
#5and maybe a follow up question (to measure the false negatives) Do you check the applicants who were denied based on their test and see where they ended up working at. E.g. you are a mid tier start up who rejects someone who ends up working at amazon as a high level engineer – do you mark that a failure?
I'd be careful to call that out as a negative; if the culture fit wasn't right, and the candidate would have been a net negative to the team, it shouldn't matter where they end up next, unless (of course) the candidate that was actually hired ends up being an even worse fit (ergo the need to fix your hiring process).
On the other hand, "they would not have been a good fit" sounds suspiciously like a blanket, non-falsifiable denial of failure. I other words, bullshit.
Re: Ask HN: Measuring the long-term benefit of interview code tests?
#6and maybe a follow up question (to measure the false negatives) Do you check the applicants who were denied based on their test and see where they ended up working at. E.g. you are a mid tier start up who rejects someone who ends up working at amazon as a high level engineer – do you mark that a failure?
I'd be careful to call that out as a negative; if the culture fit wasn't right, and the candidate would have been a net negative to the team, it shouldn't matter where they end up next, unless (of course) the candidate that was actually hired ends up being an even worse fit (ergo the need to fix your hiring process).
> I'd be careful to call that out as a negative; if the culture fit wasn't right, and the candidate would have been a net negative to the team, it shouldn't matter where they end up next
I'd be careful to presume you can know these things from an interview.
> unless (of course) the candidate that was actually hired ends up being an even worse fit (ergo the need to fix your hiring process).
Total lack of self awareness in the corporate world really is an amazing thing to behold. I suppose this is "iterating" (in HR speak, not code speak): taking a set of criteria which generates a wrong conclusion, and then applying all that to ancillary things to find more wrong answers.
Re: Ask HN: Measuring the long-term benefit of interview code tests?
#7and maybe a follow up question (to measure the false negatives) Do you check the applicants who were denied based on their test and see where they ended up working at. E.g. you are a mid tier start up who rejects someone who ends up working at amazon as a high level engineer – do you mark that a failure?
We would mark that as a success. Someone good enough to get into a top tier place, wouldn't last a year at our shop. :)
Re: Ask HN: Measuring the long-term benefit of interview code tests?
#8I don't know if my current company does, but when I first implemented them for a company I worked for ~15 years ago we definitely did.
At that company (which was a ~200 engineer, privately held, software company) we found a few things: - in person tests were less predictive than take home tests. - tests that did not provide automated test cases as examples were less predictive than those that did. - there was virtually no predictive power to 'secret test cases' that we ran without providing to the candidate. - no other part of the interview pipeline was predictive at all. Not whiteboarding, not presenting, not personality interviews, not culture fit testing, not credentials, or where experience came from, nothing. That was across all interviewers and candidates.
A few caveats about this: - this was before take home testing had become widespread and many companies screwed it up. At the time we were doing this it was seen as novel and interesting by candidates, not as just one more painful hoop they had to jump through. - we never interviewed enough candidates to get true statistical relevance. - false negatives were our biggest concern, they are extremely hard to measure (and potentially open yourself up to lawsuit). The best we ended up doing was opening up our pipeline to become less selective to account for it. This did not seem to reduce employee quality.
In a more meta-sense, that experience led me to believe that strict hiring pipelines are largely not useful. Bad candidates still get through and good candidates don't. Also, many other things have a much bigger outsized impact on productivity than if a candidate was 'good'. It turns out, humans do not produce at consistent levels all the time and things outside of what you can interview for make more impact (company process, employee health, life events, etc. all have way more impact on employee productivity than their 'score' at interview time).
Re: Ask HN: Measuring the long-term benefit of interview code tests?
#9Earlier quoted context omitted.
I'd be careful to call that out as a negative; if the culture fit wasn't right, and the candidate would have been a net negative to the team, it shouldn't matter where they end up next, unless (of course) the candidate that was actually hired ends up being an even worse fit (ergo the need to fix your hiring process).
On the other hand, "they would not have been a good fit" sounds suspiciously like a blanket, non-falsifiable denial of failure. I other words, bullshit.
See also: "you don't have enough experience," one which I most recently heard myself after four interviews and a technical assessment, in which my (passing) solution included a bugfix to the test itself.
Re: Ask HN: Measuring the long-term benefit of interview code tests?
#10We don't use coding tests in this way. We use coding tests as a screening process to ensure the candidate is in the correct ballpark.
If we are recruiting a senior, we would expect them to easily complete basic technical tests. If they are more junior we might use them only as an indicator of their ability.
I don't particularly expect a strong correlation between how well they did in the tests and their long-term ability since their value is made up of many things, only one of which is their ability in the tests.