People suck at technical interviews

There are many discussions here on HN about company hiring procedures. Company hiring procedures and their effectiveness is a heavily researched topic in industrial and organizational psychology, but most hiring managers and most job applicants haven't looked up much of the research. After reading the blog post kindly submitted here, I'll make some comments on its tl;dr summary at the end of the post.

1. many interview techniques test skills that are at best irrelevant to real working life

Yep, that's why you want your company hiring procedures to be based on research and what really matters for finding good workers.

2. you want somebody who knows enough to do the job right now

That's the ideal. That's why work-sample tests are, by replicated research, a very good hiring procedure, one of the best possible hiring procedures.

3. or somebody smart and motivated enough that they can learn the job quickly

Yep, and that's why tests of "general mental ability" are also a very effective hiring procedure, although there are some legal requirements surrounding use of those that you have to be careful about in the United States.

4. you want somebody who keeps getting better at what they do

For sure, as that is the only way your company can meet new challenges as they come up in the company's business.

5. your interview should be a collaborative conversations, not a combative interrogation

I'm not sure that the author here has provided evidence for the "should" statement in this summary, although I actually don't disagree as a matter of how I do job interviews.

6. you also want somebody who you will enjoy working with

Basically, almost all hiring managers fall victim to overemphasizing likability and underemphasizing ability to get the job done, but, yeah, you don't want to hire someone who makes the workplace miserable--that might cost you losing other good workers.

7. it's important to separate "enjoy working with" from "enjoy hanging out with"

Absolutely. The best worker in your company may not be the same person you go out with socially after work.

8. don't hire assholes, no matter how good they are

The trick here is to figure out how much annoying behavior qualifies a person as an "asshole" in a particular context, and that is not easy.

9. if your team isn't diverse, your team is worse than it needed to be

There is an increasing body of research to back up this idea.

10. accept that hiring takes a really long time and is really, really hard

Hiring is hard. It may or may not be time-consuming, depending on how efficiently you do it.

The review article by Frank L. Schmidt and John E. Hunter, "The Validity and Utility of Selection Models in Personnel Psychology: Practical and Theoretical Implications of 85 Years of Research Findings,"[1] Psychological Bulletin, Vol. 124, No. 2, 262-274 sums up, current to 1998, a meta-analysis of much of the huge peer-reviewed professional literature on the industrial and organizational psychology devoted to business hiring procedures. There are many kinds of hiring criteria, such as in-person interviews, telephone interviews, resume reviews for job experience, checks for academic credentials, personality tests, and so on. There is much published study research on how job applicants perform after they are hired in a wide variety of occupations.[2]

EXECUTIVE SUMMARY: If you are hiring for any kind of job in the United States, with its legal rules about hiring, prefer a work-sample test as your hiring procedure. If you are hiring in most other parts of the world, use a work-sample test in combination with a general mental ability test.

The overall summary of the industrial psychology research in reliable secondary sources is that two kinds of job screening procedures work reasonably well. One is a general mental ability (GMA) test (an IQ-like test, such as the Wonderlic personnel screening test). Another is a work-sample test, where the applicant does an actual task or group of tasks like what the applicant will do on the job if hired. (But the calculated validity of each of the two best kinds of procedures, standing alone, is only 0.54 for work sample tests and 0.51 for general mental ability tests.) Each of these kinds of tests has about the same validity in screening applicants for jobs, with the general mental ability test better predicting success for applicants who will be trained into a new job. Neither is perfect (both miss some good performers on the job, and select some bad performers on the job), but both are better than any other single-factor hiring procedure that has been tested in rigorous research, across a wide variety of occupations. So if you are hiring for your company, it's a good idea to think about how to build a work-sample test into all of your hiring processes.

Because of a Supreme Court decision in the United States (the decision does not apply in other countries, which have different statutes about employment), it is legally risky to give job applicants general mental ability tests such as a straight-up IQ test (as was commonplace in my parents' generation) as a routine part of hiring procedures. The Griggs v. Duke Power, 401 U.S. 424 (1971) case[3] interpreted a federal statute about employment discrimination and held that a general intelligence test used in hiring that could have a "disparate impact" on applicants of some protected classes must "bear a demonstrable relationship to successful performance of the jobs for which it was used." In other words, a company that wants to use a test like the Wonderlic, or like the SAT, or like the current WAIS or Stanford-Binet IQ tests, in a hiring procedure had best conduct a specific validation study of the test related to performance on the job in question. Some companies do the validation study, and use IQ-like tests in hiring. Other companies use IQ-like tests in hiring and hope that no one sues (which is not what I would advise any company). Note that a brain-teaser-type test used in a hiring procedure could be challenged as illegal if it can be shown to have disparate impact on some job applicants. A company defending a brain-teaser test for hiring would have to defend it by showing it is supported by a validation study demonstrating that the test is related to successful performance on the job. Such validation studies can be quite expensive. (Companies outside the United States are regulated by different laws. One other big difference between the United States and other countries is the relative ease with which workers may be fired in the United States, allowing companies to correct hiring mistakes by terminating the employment of the workers they hired mistakenly. The more legal protections a worker has from being fired, the more reluctant companies will be about hiring in the first place.)

The social background to the legal environment in the United States is explained in various books about hiring procedures,[4] and some of the social background appears to be changing in the most recent few decades, with the prospect for further changes.[5]

Previous discussion on HN pointed out that the Schmidt & Hunter (1998) article showed that multi-factor procedures work better than single-factor procedures, a summary of that article we can find in the current professional literature, for example "Reasons for being selective when choosing personnel selection procedures"[6] (2010) by Cornelius J. König, Ute-Christine Klehe, Matthias Berchtold, and Martin Kleinmann:

"Choosing personnel selection procedures could be so simple: Grab your copy of Schmidt and Hunter (1998) and read their Table 1 (again). This should remind you to use a general mental ability (GMA) test in combination with an integrity test, a structured interview, a work sample test, and/or a conscientiousness measure."

But the 2010 article notes, looking at actual practice of companies around the world, "However, this idea does not seem to capture what is actually happening in organizations, as practitioners worldwide often use procedures with low predictive validity and regularly ignore procedures that are more valid (e.g., Di Milia, 2004; Lievens & De Paepe, 2004; Ryan, McFarland, Baron, & Page, 1999; Scholarios & Lockyer, 1999; Schuler, Hell, Trapmann, Schaar, & Boramir, 2007; Taylor, Keelty, & McDonnell, 2002). For example, the highly valid work sample tests are hardly used in the US, and the potentially rather useless procedure of graphology (Dean, 1992; Neter & Ben-Shakhar, 1989) is applied somewhere between occasionally and often in France (Ryan et al., 1999). In Germany, the use of GMA tests is reported to be low and to be decreasing (i.e., only 30% of the companies surveyed by Schuler et al., 2007, now use them)."

[1]

http://mavweb.mnsu.edu/howard/Schmidt%20and%20Hunter%201998%...

[2]

http://www.siop.org/workplace/employment%20testing/testtypes...

[3]

http://scholar.google.com/scholar_case?case=8655598674229196...

[4]

http://books.google.com/books?hl=en&lr=&id=SRv-GZkw6TEC

[5]

http://intl-pss.sagepub.com/content/17/10/913.full

http://www.economics.harvard.edu/faculty/fryer/files/Fryer_R...

[6]

http://geb.uni-giessen.de/geb/volltexte/2012/8532/pdf/prepri...

Having smart people try to determine if another person matches their same definition of "smartness" is fraught with peril. There are a dozen dimensions to "smartness," and not everybody aligns properly.

It's similar to the quote "You have to be twice as smart to fix broken code as you were when you wrote it." Extremely clever code with bugs is provably unfixable. Extremely clever interviews are almost provably bad filter criteria.

But, in the context of interviewing, evaluating _a person_ isn't the same as evaluating _their immediate output_. Evaluating the future output of a person isn't the same as evaluating "can this person replicate intro-to-CS exercises they haven't touched in 12 years?"

Correct interviews require, gasp, strong compassionate people skills in addition to domain knowledge where you can challenge candidates. You've always got to figure out what they actually know versus what they think they know versus what they say they can do versus what they can actually do.

Then there's an entire other issue of "Smart People" versus "Capable People." Most people in power end up being "smart," but not necessarily capable any longer (by "failing upward" and now having magic protections). Some people end up being decision makers with little actual creation responsibilities (read: anything they could actually be judged against), so they are free to just be amazing with little detriment for their decisions. But, sometimes you need a number of "smart but not capable" people to balance out half a company being head-down technically but not necessarily aware of larger issues plaguing them. (Did I just invent managers?)

But, trying to get management and HR to change from the conventions is extremely difficult. I have not been successful.

I would go a little further than Laurie does. I think several of the goals he sets up for his process are not in reality achievable in an interview process.

Starting axiom: job interviews are among the most hostile experiences professionals endure in our industry. I think back to my own interviews, and compare them to public speaking, professional disasters, death march projects with insane deadlines, intractable politics, and it's pretty much the interviews alone that increase my heart rate. For the past two years, I've tried to make an effort to peek in on the interviews we do here at Matasano, and what I've seen corroborates the belief. Several of our best hires were physically shaking during their interviews. And we try to be nice! In no other common situation does a tech worker find themselves interrogated by a panel of strangers whose implicit goal is to knock them out of contention.

Given that interviews are a psychologically challenging experience, and thinking about things like the concept of "depletion" of ego or willpower, it's straightforward to see some severe limitations to what can be accomplished in an interview setting. If you're spending lots of energy trying to keep from jumping out of your skin in an unpleasant situation, it's much harder to solve problems that themselves require lots of energy.

Past that, a hypothesis, which is unpleasant for some tech workers to hear (cold comfort: I'm 100% certain it applies to me as well). Software developers are not, as a cohort, particularly effective intuitive psychologists. Virtually none of us have any training in the subject. We tend sharply towards introversion. We train our brains day-in and day-out by repeating tasks that do nothing to developer our ability to read in-person social cues. For that matter, we tend as a group to eschew forms of communication in which tone of voice, body language, and emotional cues are even transmitted!

But several of the objectives Laurie sets out demand exactly that kind of analysis. "Can the candidate intelligently discuss technology?" Well, that's subjective, and worse, vague and abstract. Laurie tries to nail "intelligently" down, but I think we can all see that there are other ways in which someone can be "intelligent" about technology that evade those criteria. Since we all intuitively know that, we substitute our own cognitive biases for "intelligently". All of the sudden, we're gauging "confidence" and "comfort level"... we've decided to be psychologists instead of engineers.

So, two changes I would urgently suggest Laurie consider for his process:

* Audit the whole process for tasks that could generate false positives from a nervous candidate. You aren't interviewing people to determine how good they are at interviewing, because interviewing doesn't generate money for your company. Try to build a process that is immune to discomfort and lack of confidence. It can be done! Another thing that we've found very effective: "prime" candidates early and repeatedly with non-adversarial conversations that aim to disarm them. We start our whole (multi-week) process with an hour-long version of this. We also try to innoculate our process by communicating in as much excruciating detail as we can what it will entail.

* Eliminate all subjective questions and standardize what you're left with. Engineers, in my experience, fucking hate this. But it's the right thing to do: ask every candidate, as much as possible, the same set of questions. We have a question structure that minimizes open-ended questions but has some structured "exercise" questions that give the candidate some freedom to move around --- the results of those questions can be put on a scoresheet and, to some extent, compared apples-apples to other candidates.