The Terrible Technical Interview

I have found the most reliable measure for a candidate is looking at their public interactions/contributions. Seeing what projects they choose to work on in their spare time, how they speak at user groups, how they interact on an "issue" online. Blog posts are also great. For me, ability to communicate is such a huge part of being a successful developer, that being able to see how they communicate is always a very strong signal of intelligence, mastery, and interpersonal style.

So this works great for people that partake in such things, but there are clear still many great developers (perhaps the vast majority) that don't have this type of extensive public profile. These are the applicants that I fear false-negatives for the most, and the candidates for which I am not confident of my interview techniques.

Could it be that the proper way to technically interview such a candidate is to offer them a "fellowship" to work on an open-source project? Point them at any project (used by the company or a favorite project of their own) that they'd like to contribute to, and offer them a small stipend and enough time to make a legitimate contribution to open source? And use that as the technical portion of the interview?

I think the challenge here will be in the overhead of finding and evaluating suitable projects, given that you can use each one only once.

What this process seems to be tuned for excluding potential candidates. And if the end result of many many interviews is the idea that the talent pool outside silicon valley is poor, perhaps that reflects the focus of the approach.

I suspect that it is very likely that this process ends up excluding very high-level producers.

I've found this technical interview process to work very well. This is done after the initial phone screen.

1. Show them a problem with your product along with the code. For front end developers it could be how form invalidation errors are being presented to users. 2. Ask them to figure out what why the code is doing this and observe them troubleshoot the code. 3. Tell them to fix the problem in the code and observe them apply a fix, test, and debug it. 4. Ask them to architect a better solution to the problem and to explain what makes it better. What would would be drawbacks to their solution.

The benefits of this approach is that you can evaluate directly how someone solves problems, not how well they communicate, nor how knowledgeable they are. It also helps me judge how fast they are. Sometimes I ask people to estimate how long it would take to solve this and time them. The last step helps establish how they think through their solutions and how well they can communicate their ideas.

Additionally, I get to see how quickly they might get up to speed with our codebase and I can hear other solutions to some of the technical problems we are facing which is incredibly beneficial as a small startup.

I remember back when I was a teenager looking at colleges seeing a private tech college with a high acceptance rate and a public state college with a low acceptance rate. The Princeton Review had rated the private tech college as much more selective than the public state college. Teenage me was confused - how was that possible? - the state school had a significantly lower acceptance rate!

The reason is, of course, self-selection bias. On average, the people who applied for the tech college were just stronger students than the people who applied to the state college. This likely is because applicants to the tech college were more likely to be people whose primary goal was to peruse an academic discipline that interested them while the state school had a good number of students applying who wanted to party and attend football games. If you were to look at students who applied to BOTH schools, you would actually see a lower acceptance rate for the tech school than the state school.

I suspect that the same factor is at play here. I suspect that the average quality of a candidate who applies for a software security engineering position is much higher than the average quality of a candidate that applies for a enterprise software development position. The software security engineering position is an esoteric position that is more likely to attract applicants who are enthusiasts or at least very interested in the field. The enterprise software development position is more likely to attract anyone with a tech background looking to phone it in and collect a paycheck.

If this is true, while both Thomas and Steve have a difficult time hiring developers, they have a difficult time for fundamentally different reasons. Thomas is like the tech school - his difficulty is with getting candidates to apply. Steve is like the state school - his problem is with separating qualified candidates from "phone it in and collect a paycheck" candidates.

This is why I am skeptical of Thomas thinking he's got a much better way of hiring figured out. Maybe he does for companies that focus on security - judging by his Linkedin profile, he's only worked for companies focused on security. This is certainly valuable, but he has no idea what it is like for Steve.

By the way, I think many of us have seen this self-selection bias at play even within our own companies. Post a primarily Java job and a primarily Scala job and then try to tell me that the same principles apply for filling both positions.