Why we don't hire programmers based on puzzles and tricks

I doubt that constitutes a majority of the people you pass on, but it's definitely going to be double digits.

I've never hired using a white board coding test, and out of 30 or 40 folks i've had to let 2 go pretty quickly because it was clear they couldn't do the work, but the rest did as well as somebody who would have passed one of your quizes - I'd expect we'd see similar long term failure rates in both sets.

I'm sure it makes sense at google where the hiring procedures don't leave much room for error, but I'm sure I've been able to hire some great folks that you would have missed. Arguably the best heads down brilliant engineer I've hired (well tied for #1) completely melted down on his first interview. Tried again with drinks and a show at Yoshi's and he did so well I would have hired him to be my boss. He easily pulled 3x his own weight for us and was very loyal and easy to work with.

So many people go on and on about how hard it is to hire yet they go around systematically rejecting all the same candidates for all the same reasons.

My experience with Amazon was actually the opposite, but I might have been lucky. All the interviewers were very friendly and started from simple questions and went on to more complex ones. Never a "do you know concept a from language Y". In fact, one of them worked with me through the solution to a pretty tricky problem, and when we were done, he actually took a picture of my solution on the whiteboard! So yeah, it was fun :)

It depends on the "problem solving process" you are trying to get insight into.

These procedural "how does the candidate get to the answer?" questions are great for procedural thinkers. Even if they don't know details that feed into reaching an answer, they can show you how they'd get there.

Unfortunately, if your staff ends up entirely composed of procedural thinkers, you may find your company challenged at innovation. For that, you need candidates good at "Eureka" moments.

Your Eureka candidates will often do quite badly in the procedural "what's your thought process?" questions in the one hour interview context. Measured over a 24 hour period, they may do notably better.

"The problem solver initially has a low probability for success because they use inappropriate knowledge as they set unnecessary constraints on the problem. Once the person relaxes his or her constraints, they can bring previously unavailable knowledge into working memory to solve the problem." -- http://en.wikipedia.org/wiki/Eureka_effect#How_people_solve_...

What you're looking for here is both broad and deep domain expertise, coupled with an ability to understand the question well then let the subconscious mind pull together the insights needed for a breakthrough solution. This process cannot be talked through out loud. The answer may take longer (hours, a day or two, probably reached in the proverbial shower), but will often be an answer that gives you a long term competitive edge over a blunt procedural approach.

This said, it's not clear to me how to interview for this ability. I'd start by considering a way to understand the depth and breadth of a candidate's exposure to technology and domain knowledge. This is the opposite of interview processes using dives into the interviewer's area of expertise. I might consider a two part interview a few days apart, where the candidate gets the context in part one, and shares their thinking in part two.

> Asking a candidate a difficult brainteaser that may be even close to impossible can be extremely revealing. There isn't a canned response, you get to see them work under pressure, you experience live problem solving and you can understand how they move through their work better. All of these factors are just as important as competency when hiring.

Except that this situation is an artificial construct. You're covering their mouth and nose and saying, "Now show me how you breathe." You are favoring candidates whose problem solving is informed by the belief they can reach an emergency answer on their own through brute force, rather than those whose problem solving starts with the assumption that someone else has already solved or mostly solved the problem and who is great at finding the "shoulders of giants" to stand on, building an informed answer a priori.

I'm reminded of getting my deep water scuba certification. You're not interviewed for a diving salvage job by repeatedly seeing how you handle having your breathing apparatus turned off. Yes, a diver should be able to "perform under pressure", but what you really want are divers who help ensure nobody on the team ends up under pressure.

Some are just way to shy and never ask question. That's a slight negative. I am shy too, but I need to get the job done I'll go find out who to talk to, ask question, figure out specs, see who's done what before and so on. That is an important trait.

I'll give out hints as well, it is not too structured. I don't immediately subtract some point or anything. In fact I might hold it against them if they don't ask questions.

I disagree that programming puzzles don't measure what they're supposed to. Programming puzzles are a microcosm of what programming entails day-to-day. They test the same exact mental faculties. Sure, no one needs to write strcmp using primitives anymore, but the task of doing so exercises the same mental processes that are involved in orchestrating the various components when handling a web request. Doing these type of "low level" tests then allow you to include components that do measure one's mental flexibility and creativity, i.e. intelligence.

Sure, we can give a candidate a test that exactly measures what they'll be doing on the job. But what if the job changes? What if we need them to use a language or a framework outside of their comfort zone? This is the reason these tests are not specific to particular tools. You want a programmer that has the intelligence and command of computer science concepts to quickly jump between various tools with little effort. Comparing strings and reversing linked lists do absolutely measure this capability. No, its not testing whether you memorized how to do these things, its testing whether you have the mental flexibility to figure it out on the spot.