Live data from Hacker News

Ask HN: Measuring the long-term benefit of interview code tests?

news.ycombinator.com

21–30 of 76 posts

Re: Ask HN: Measuring the long-term benefit of interview code tests?

#21

For developers, coding tests that include deployment / infrastructure components (i.e., deploy your solution to a cloud container, or, build and compile your solution for desktop platform testing) are uniformly consistent with long-term impact / success. Problem solving at the algorithmic layer may be inversely correlated to success, if a candidate lacks a production skill set. Unless one's focus is research and deve…

With the original question being about measuring correlation between interview performance and actual long-term performance on a job, I'd love to see the numbers you are basing your opinion on regarding testing for "production skills" and long-term job performance.

Re: Ask HN: Measuring the long-term benefit of interview code tests?

#22

Earlier quoted context omitted.

We tried that at one company I worked for and it worked well enough. Our contract with the consulting firm said if we dumped the contractor within 90 days we didn't pay a cent for any of their time. This resulted in the consulting firm only sending us candidates that had good reviews in prior engagements. And good reviews from prior engagements strongly correlates with good reviews in future engagements.

Can you tell me which consulting company was that. We have a few contract positions where we can use consultants .

I want to say it was TEKSystems. Some caveats here. We may have had to sign an exclusive deal with them, that we would not hire through anyone else. Also, we were a big client with an extremely good relationship with them, they made a lot of money from us. Also, it could've been 30 or 60 days on the limit. But the concept was in place.

Re: Ask HN: Measuring the long-term benefit of interview code tests?

#23
I would say anything having indirect correlation has no easy way to measure. Ultimately a company is either looking for product market fit, customer growth or revenue/profit/cash flow depending which stage the company is in.

On top of being hard to measure, the data points generated through hiring is just too few and the data collection process is too long and subjective

Just ask your team if they like the new hire, can they make progress together. Things like do you like working with the new hire? Is the new hire bringing in new insights to the team? Is the new hire easy to work with? Is the new hire learning new things.

And most importantly, can the team let go of mismatch fast enough. Overall I would say it is just not worth it in measuring hiring.

Re: Ask HN: Measuring the long-term benefit of interview code tests?

#24

I don't know if my current company does, but when I first implemented them for a company I worked for ~15 years ago we definitely did. At that company (which was a ~200 engineer, privately held, software company) we found a few things: - in person tests were less predictive than take home tests. - tests that did not provide automated test cases as examples were less predictive than those that did. - there was virtual…

Very interesting. Question:

How did you measure the candidate once hired?

What factors were indicative of a "good" hire vs. a "bad" hire?

Re: Ask HN: Measuring the long-term benefit of interview code tests?

#25

For developers, coding tests that include deployment / infrastructure components (i.e., deploy your solution to a cloud container, or, build and compile your solution for desktop platform testing) are uniformly consistent with long-term impact / success. Problem solving at the algorithmic layer may be inversely correlated to success, if a candidate lacks a production skill set. Unless one's focus is research and deve…

Are you saying this because you measured it, because you have anecdotal evidence that it is true, or because it is something that you believe?

Re: Ask HN: Measuring the long-term benefit of interview code tests?

#26
The big FAANG do for what it's worth. They have entire ML pipelines looking at hiring. The following isn't about interview effectiveness but is one example of the analysis done:

https://catonmat.net/programming-competitions-work-performan...

Re: Ask HN: Measuring the long-term benefit of interview code tests?

#27

For developers, coding tests that include deployment / infrastructure components (i.e., deploy your solution to a cloud container, or, build and compile your solution for desktop platform testing) are uniformly consistent with long-term impact / success. Problem solving at the algorithmic layer may be inversely correlated to success, if a candidate lacks a production skill set. Unless one's focus is research and deve…

As a developer with an arts background (studied music, happened into tech work after college through years of mostly self-taught hobbyist tendencies) I'd agree that the design of these tests themselves is a factor that's thoroughly understated.

Timed algo challenges encompass a slew of antipatterns in terms of how good code is actually written and shipped. To begin with, pitting someone against a clock and hidden test cases (and a foreign editor) is actively optimizing against solutions that are readable to other human beings -- or to the person writing them, a year from now. The nature of running them in a browser means that it can't evaluate a person's capacity to actually use tools outside of core language functionality. Never mind that building the entire exercise around predetermined test cases precludes any way to gauge whether the person taking it has an understanding of writing tests.

And that's assuming your test environment doesn't add obnoxious and arbitrary restrictions of its own. Like telling you that using documentation is cheating. (Btw imagine listening for ctrl+t here, but not ctrl+n.) Or offering you "the language of your choice," but then throwing API call exercises at you while limiting your choice of JavaScript runtimes to a bare installation of Node -- the only one still in active development, out of a list that also includes every browser you would use to actually access the test -- that doesn't support Fetch.

Re: Ask HN: Measuring the long-term benefit of interview code tests?

#28
Not empirically, but my manager focuses primarily on the engineering expensive for our team and potential hires. This results in explicit feedback gathering, modifying our process accordingly.

We have short, standardized, broad interviews. We look for what can be added to the team rather than poking holes, and we're still trying to improve.

Re: Ask HN: Measuring the long-term benefit of interview code tests?

#29
post #24

I don't know if my current company does, but when I first implemented them for a company I worked for ~15 years ago we definitely did. At that company (which was a ~200 engineer, privately held, software company) we found a few things: - in person tests were less predictive than take home tests. - tests that did not provide automated test cases as examples were less predictive than those that did. - there was virtual…

Very interesting. Question: How did you measure the candidate once hired? What factors were indicative of a "good" hire vs. a "bad" hire?

We compared their performance review scores. I was always leery of this given how fraught performance reviews are, but that’s how the company judged employee ‘worth’ so it made sense to align there.
Post reply on HN