The Utter Uselessness of Job Interviews

Making sure I understand you: When you refer to work sample tests here, that refers to the crypto challenges and things like that that you published at Matasano? And you're saying that was much more predictive than the list of facts methods, right?

The biggest problem with instinct/intuition is that you need to have enough relevant experience for it to work. Your intuition ends up being completely wrong when you brain doesn't have enough data to work with. When it does have enough data it's an extremely powerful tool.

I think it would be cool if a company like Google published anonymized data about the correlation between interview "scores" and some post-hire metrics. My guess is the correlation would be poor to non-existent. Picking those metrics is difficult, perhaps an obvious one that is difficult to game is how long the employee stayed with the company. Another interesting one would be anonymous peer evaluations where it's guaranteed no one would see the data points so it wouldn't suffer from the problems 360 reviews have.

I think the best you can do is some sort of mix between a more structured portion trying to gauge where the candidate is in terms of knowledge and intelligence and a less structured portion where you try and get more of a "feel" for the candidate's personality. You have to realize that under the best of times it's not perfect. Rejecting someone can't code at all for a software position should be relatively easy. Ability to tackle bigger things can be gauged by looking into previous projects and through references. I think there's a big region though where the outcome is difficult to determine.

I think back to the very first time I was involved in a hiring decision. The guy was very smart, technically capable, engineering degree; PhD material. Seemed pleasant enough. Got hired and IMO definitely the kind of person a company like Google would hire with their process. He lost interest fairly quickly on the job. Working with him I found he had some very odd, not to say crazy, political opinions. Everything was too boring for him so he didn't really get that much done. Couldn't really work independently at all. He left the job, left the country, and I think he ended up being a cab driver. Not sure. Yet another example is someone fresh out of school with a CS masters degree who despite all the help of the team could simply not wrap his head around the project and become productive. On paper all the right credentials but first real world job and he couldn't cope for some reason. Ended up leaving. I've seen a few CS background people just not find their place. I'm sure we've all seen situations where we wonder how some person got hired and how come they're still there. Over time I think I learned to do a better job hiring/interviewing and have had a stream of pretty successful hiring decisions.

It turns out that it takes many months on the job to really know the fit. Even if the person is capable the specific job or team may not work out. Some people are very good at making it look like they're accomplishing things while they're not really. There's no way any company has a secret sauce of only hiring "great" people and even great people will do poorly under certain circumstances and conversely not so great people can be very successful under the right circumstances. Some people can grow really quickly while others can't.

I didn't say it was based on hard data, that's why it's a hypothesis...

But, if you really believe that it's a huge leap to hypothesize that interviews would be a better measure of subjective social skills than metrics like GPA, I don't know what to tell you.

> It's gut/instinct thing. If you have a mechanical approach, anyone can game the system and get a job because humans can be like chameleons to present themselves as the right candidate, and they can study for the interview. Only way IMO is to have that 3rd eye or whatever you call it... instinct, gut feeling, etc.

This is ridiculous. Mechanical approaches can be gamed, but so can gut instinct. You just have to look at the White House to see that. A lot of gut-instinct voters got hoodwinked by a skilled self-promoter. And he's a gut-instinct guy himself, so he's getting led around by whoever's got his ear.

Gut instinct, though, is worse in several ways. One is that it's not transmissible. As a company grows, how do you scale? Another is that it's not necessarily repeatable. Was that person bad, or is your gut off because you're tired, depressed, or upset? A third is that it can't be consciously improved. If your mechanical process has a flaw, your team can discuss it and come up with solutions. But if your gut judgment isn't good enough, what can you do?

I am a big fan of gut instinct as a component of a hiring process. Often, thanks to experience, we perceive things we have a hard time articulating, and it's wise to pay attention to that. But I think it's a giant mistake to treat our subconscious as some sort of mystic oracle that we must worship and never question.

Google's hiring process is famously painful, and they lose a lot of great candidates that way. I wouldn't trust their insights to build a hiring pipeline.