Technical interview performance is kind of arbitrary

The headline is "interview performance is kind of arbitrary," but the data solution proposed in the article is "interviewers rate interviewees in a few different dimensions," which is not any less arbitrary.

I appreciate there is an appendix addressing this issue, but it does not absolve the issues the analysis, especially since the appendix uses a "Versus Rating" to justify the statistical accuracy of the system, which is also calculated somewhat arbitrarily (since the Versus Rating is derived from the calculated interview score, wouldn't it be expected that the two have a relationship?)

The fact that the results of the non-arbitrary score are centralized around 3 out of a 4 max (instead of the midpoint of 2) implies a potential flaw or bias in the scale criteria. (The post notes that people who get a 3 typically move forward; maybe selection bias is in play since companies would not interview unskilled people in the first place)

That's not to say that the statistical techniques in the analysis themselves are unimpressive though. I particularly like the use of FontAwesome icons with Plot.ly.

Not too surprising when you consider there isn't really a standardized guideline and every interviewer asks questions of varying difficulty. Sometimes interviewers don't even ask candidates the same question and tailor them based on the candidate's resume and experiences. I've had interviews as simple as writing a function that outputs if two strings are anagrams, and other interviews that test dynamic programming knowledge and other interviews that tested my knowledge of concurrency. At the end of the day, its luck of the draw which interviewer you get and what questions he decides to ask you.

Most interviewers don't ask enough technical questions to have any idea what a candidate knows or doesn't know. If their one or two questions happen to be something the candidate knows well, they'll call them a genius. If they happen to not know, they'll label them an idiot.

You can learn a lot more from 20+ rapid fire questions than forcing a candidate to eek out an answer to something they're not familiar with. And once you establish the areas they're familiar with, you can ask them truly useful questions.

The key is to look for people who have strengths and not worry at all about gaps in their knowledge. Anyone who has earned genuine expertise in one area will be able to do so in other areas.

The other big mistake most interviews make is forgetting about the "Curse of knowledge" https://en.wikipedia.org/wiki/Curse_of_knowledge

I've seen people research the answer to a question before an interview and then expect candidates to be equally informed without that advantage.

Good, because I suck at technical interviews, but built my own SaaS offering (working the corporate job until my customer base is large enough).

I wonder how many employ Kahneman's recommendation based on his book, "Thinking, Fast and Slow":

> Suppose that you need to hire a sales representative for your firm. If you are serious about hiring the best possible person for the job, this is what you should do. First, select a few traits that are prerequisites for success in this position (technical proficiency, engaging personality, reliability, and so on. Don't overdo it — six dimensions is a good number. The traits you choose should be as independent as possible from each other, and you should feel that you can assess them reliably by asking a few factual questions. Next, make a list of those questions for each trait and think about how you will score it, say on a 1-5 scale. You should have an idea of what you will call "very weak" or "very strong."

> These preparations should take you half an hour or so, a small investment that can make a significant difference in the quality of the people you hire. To avoid halo effects, you must collect the information on one trait at a time, scoring each before you move on to the next one. Do not skip around. To evaluate each candidate add up the six scores ... Firmly resolve that you will hire the candidate whose final score is the highest, even if there is another one whom you like better — try to resist your wish to invent broken legs to change the ranking. A vast amount of research offers a promise: you are much more likely to find the best candidate if you use this procedure than if you do what people normally do in such situations, which is to go into the interview unprepared and to make choices by an overall intuitive judgment such as "I looked into his eyes and liked what I saw."