The Terrible Technical Interview • Hacker News Board

Re: The Terrible Technical Interview

#211

Post by nitrogen » Sun, Mar 22, 2015, 6:07 PM UTC

tptacek wrote: post #209

nitrogen wrote: post #207

Earlier quoted context omitted.

A quick Google search ("trunk-first vs branch-first svn") suggests that you're not using common terminology. It's not a terminology question, it's a concept question. A good candidate should be able to recognize abstract concepts regardless of the words used to describe them. Even in an interview setting.

Why? What makes being able to persuasively discuss this particular concept an attribute of a good candidate? And, when you identify the answer to that question, can you then answer: is this question the best means I have of assessing that attribute?

I agree with you that this type of technical interview is a great way to fail to hire ideal candidates. I've experienced this myself. I was commenting only on the specific complaint about "terminology," and that one should be able to see through unfamiliar terminology in an interview, not that such an interview is the best way to hire excellent developers.

Re: The Terrible Technical Interview

#212

Post by frostmatthew » Sun, Mar 22, 2015, 8:14 PM UTC

tedunangst wrote: post #206

frostmatthew wrote: post #188

Earlier quoted context omitted.

> Consider the set ( -10, 4, 4, 4, 4, 4, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 20, 50). I didn't say it was impossible to construct a set that would yield only 10% as above average, I said there "is no reasonable definition of average" - if you feel the above set accurately represents the distribution of the caliber of developers then we clearly have very different opinions of what's "reasonable."

I don't think it's too far off the mark. What do you think the distribution looks like? Or one could replace the mathematical term average with the word ordinary . Is it possible for 60% of developers to be ordinary?

> What do you think the distribution looks like?

That would depend on what set of developers we're looking at:

All developers - this will be very bottom-heavy, people [usually] get better with experience and there's obviously a lot less people that have been doing this for 20 years than having been doing it for two. Additionally people who are bad at a profession are more likely to change careers than those that are good (this is by no means an absolute, I wouldn't even go as far to say most bad engineers change professions, I'm just saying they're more likely to - further contributing to higher caliber corresponding well to years of experience).

Developers with similar experience - this is much more useful as there's not much point comparing someone who's been doing something for decades with someone on their first job. I would expect this to be a fairly normal distribution.

Developers interviewing for a particular position - applicants will largely self-select (and the initial screening process would further refine that) so this group will largely have similar experience (i.e. you're typically not interviewing someone with no experience and someone with 25 for the same job). But it won't match the previous distribution because, as someone else commented, the bad ones are looking for work more often (and for a longer period of time). Do the interviewees you wouldn't hire outnumber the ones you would? Yes, definitely. Do they outnumber them by a factor of a hundred to one? Definitely not. Ten to one? Probably not - if they do it probably represents a flawed screening process causing you to interview people you shouldn't (or not interview the people you should) rather than an indication that only one out of every ten developers are worth hiring.

Re: The Terrible Technical Interview

#213

Post by itsdrewmiller » Sun, Mar 22, 2015, 8:34 PM UTC

sytelus wrote: post #152

fsk wrote: post #5

False negatives are a lot worse than most interviewers think. (False negative = you pass on a good candidate, false positive = you hire someone who turned out to be unqualified) If a "good" candidate is a 1-in-100 find, then each false negative means you have to look at another 100 candidates. Also, if you decrease your false negative rate by more than you decrease your false positive rate, you're actually hiring MOR…

No. Many companies have hiring rate of 10% [1] which means hiring 1 candidate may cost you 10 person-days assuming full day interviews for each candidate [2]. You can probably double that to cover the cost of phone screens, so let's say 20-person day of work per good hire. Assume you had 4 false negatives just because you are so aggressive and so a good hire ended up costing you 100 person days instead. This cost is…

You didn't actually respond to his comment. Assuming, for example, you have:

* 99% of applicants are bad * 50% false negative (You look over about one good developer for every good developer you hire) * 1% false positive (one out of a hundred bad devs can snooker you into hiring)

In that scenario, you're twice as likely to hire a bad dev as a good one. And if you halve your false positive rate by increasing your false negative rate by 50%, you're still twice as likely to hire a bad dev, it will just take you twice as much work.

Re: The Terrible Technical Interview

#214

Post by SomeCallMeTim » Sun, Mar 22, 2015, 8:39 PM UTC

frostmatthew wrote: post #116

fsk wrote: post #85

Earlier quoted context omitted.

Even if you set the "good" percentage to 10%, too high a false positive rate will still ruin your results. Based on the people I've worked with over the years, I say that the actual skill distribution is: 5% toxic - These are the people who will ruin your business while deflecting blame to other people. 25% subtractors - These are the people who need more attention and help than the amount of work they get done. In t…

> 9% above average There is no reasonable definition of average that would only allow for 9% above that (or 10% including the 1% you marked as brilliant). Average is usually considered as either the 50th percentile (in which case you would have ~50% above this) or some middle range (e.g. 25th - 75th percentile). Since you said 60% are average we'll consider an appropriate range as average, the 20th - 80th percentile.…

I'm not OP, but it feels like you're arguing semantics. YES, that's the technical definition of "average," no argument, but I don't think he/she meant mathematically average.

If you substitute with these terms:

- 5% toxic

- 25% subtractors

- 60% competent

- 9% exceptional

- 1% brilliant

...then there's no reason to apply (or defend!) the mathematical definition of "average." And I think those numbers actually seem somewhat reasonable, based on my own exposure to working developers in various industries. What this doesn't count is the the "FizzBuzz effect," where ~95% of the people who are interviewing at any one time (in a tight market) tend to be from the bottom end of the spectrum.

Even within the broader pool of programmers, the line between subtractors and competent is very project-dependent, in my opinion. For some levels of project complexity, the line might actually invert to 60% subtractors and 25% competent, while for far less complex projects, it might be 5% subtractors to 80% competent.

In the former case I'd want an exceptional developer, while in the latter the exceptional developer probably wouldn't even apply, or would quit out of boredom.

Re: The Terrible Technical Interview

#215

Post by jimbobimbo » Sun, Mar 22, 2015, 9:09 PM UTC

jghn wrote: post #195

jimbobimbo wrote: post #108

I've been on both sides of the process and in both cases I prefer the whiteboard. As an interviewer, I simply don't have an hour or two to evaluate someone's project. I want to see a strong resume and ask a few questions to understand whether the resume is real or not. A simple whiteboard question or two is great for that. You can always see whether the person is capable of writing code and it always brings up questi…

"As an interviewer, I simply don't have an hour or two to evaluate someone's project." These are people with whom you'll be working together with for years. If you can't spare a few hours to make sure you are finding the right person, what does that say to a candidate?

The interview is a tool to quickly evaluate the candidate. Not every candidate reaches the point where they could be even considered being a person I'll "be working together with for years". There're more than one candidate at any given time and I still have my day job to do.

I don't know what does this say to a candidate, but I surely hope they're mature enough to understand that there's nothing personal about it. They will be treated with dignity and courteously during the whole process, but that's all I can promise.

Re: The Terrible Technical Interview

#216

Post by jimbobimbo » Sun, Mar 22, 2015, 9:10 PM UTC

jqm wrote: post #156

jimbobimbo wrote: post #108

I've been on both sides of the process and in both cases I prefer the whiteboard. As an interviewer, I simply don't have an hour or two to evaluate someone's project. I want to see a strong resume and ask a few questions to understand whether the resume is real or not. A simple whiteboard question or two is great for that. You can always see whether the person is capable of writing code and it always brings up questi…

If you don't have an hour or two as an interviewer, you probably aren't getting very good results. This is a pretty small investment of time considering what is at stake.

I actually hired two best performers at my previous company this way. The amount of people interviewed total is about 30.

Re: The Terrible Technical Interview

#217

Post by jimbobimbo » Sun, Mar 22, 2015, 9:15 PM UTC

mikekchar wrote: post #115

jimbobimbo wrote: post #108

I've been on both sides of the process and in both cases I prefer the whiteboard. As an interviewer, I simply don't have an hour or two to evaluate someone's project. I want to see a strong resume and ask a few questions to understand whether the resume is real or not. A simple whiteboard question or two is great for that. You can always see whether the person is capable of writing code and it always brings up questi…

I am going to have to disagree with you. The whiteboard question does not always work. In fact, while I am generally able to do whiteboard questions without a problem, I had an experience 2 years ago where I completely blanked and was absolutely unable to write any code on a whiteboard. If they had handed me a computer (or if I had not been too flabergasted to ask for one), I would have been fine. In the end, the com…

Mike, actually I totally empathize with your feeling towards the whiteboard questions. I would probably be flabbergasted myself if offered a live coding session. Whiteboard is much more forgiving in the interview setting, in my opinion. When I'm interviewing, I keep stressing the fact that I don't care about correct syntax, coding style and whatever have you when you are doing whiteboard - only idea counts. I also leave them for about 20 minutes to give them space to think. Not sure if that helps, but I've not seen people having problems after such intro.

"Non-job projects" != "test projects". I do have non-job projects and showcase them when interviewing, but I would not take a test project as part of the interview process, unless I have really good reasons to do so.

Re: The Terrible Technical Interview

#218

Post by politician » Sun, Mar 22, 2015, 9:36 PM UTC

tptacek wrote: post #201

politician wrote: post #135

Earlier quoted context omitted.

I completely agree with this - asking about VCS is a cross-cutting question that nearly all candidates have experience with unlike, for example, graphics, desktop, or cloud experiences. So, requesting that the candidate explain their VCS workflow is a way to generate a practical and relevant data point to use when evaluating multiple candidates. For example, the 1-in-a-million candidate that has used git with a CI/CD…

I've carefully read every comment on this subthread, and here's what I think: This question is a trifecta of ineffective candidate screening tactics: (a) It's a technical screening question, one a strong candidate could get wrong, based on a technical aptitude that is trivial to teach on the job and thus rarely worth paying a premium for. (b) It's a subjective technical question, for which reasonable engineers can ha…

Thanks for being open about your perspective on the interviewing process, and for taking the time to follow up on these threads.

Do you believe that technical phone pre-screens are ineffective in general? From what I've read Matasano doesn't pre-screen candidates, but provides complementary study materials instead. Is that because the subject matter is specialized? Would you approach hiring web dev roles differently?

From your remarks, it sounds like you would reject the practice of asking open-ended interview questions (e.g. describe your workflow, describe a typical day, describe a recent project) due to interference from the interviewer's bias. What, if any, value do you place on open-ended questions?

Re: The Terrible Technical Interview

#219

Post by fsk » Sun, Mar 22, 2015, 9:46 PM UTC

itsdrewmiller wrote: post #213

sytelus wrote: post #152

Earlier quoted context omitted.

No. Many companies have hiring rate of 10% [1] which means hiring 1 candidate may cost you 10 person-days assuming full day interviews for each candidate [2]. You can probably double that to cover the cost of phone screens, so let's say 20-person day of work per good hire. Assume you had 4 false negatives just because you are so aggressive and so a good hire ended up costing you 100 person days instead. This cost is…

You didn't actually respond to his comment. Assuming, for example, you have: * 99% of applicants are bad * 50% false negative (You look over about one good developer for every good developer you hire) * 1% false positive (one out of a hundred bad devs can snooker you into hiring) In that scenario, you're twice as likely to hire a bad dev as a good one. And if you halve your false positive rate by increasing your fals…

I see what you're saying. I was pointing out the dangers of false negatives, but he responded that he hires 10% of the people he invites for an onsite interview.

Even if you hire 10% of the candidates you on-site interview, that says nothing about your actual false positive or false negative rate. For all I know, he could be weeding out all the top candidates at the pre-interivew stage, and then hiring the best of a mediocre group of people.

It's easy to measure your false positive rate, people you are forced to fire (or wish you could fire if not for corporate bureaucracy).

It's harder to measure your false negative rate. The only way you could measure your false negative rate is to pick a random sample of people who fail your interview, AND HIRE THEM ANYWAY. (However, that could be a lawsuit risk. It would be unfair to the people who hire despite failing the interview. A small business couldn't afford to do it, only some huge corporation could do the experiment.)

Also, I doubt the ability of most businesses to identify the best performers AFTER THEY ARE HIRED and working there for a couple of years.

Re: The Terrible Technical Interview

#220

Post by jdmichal » Mon, Mar 23, 2015, 1:18 AM UTC

Is it maybe time to start allowing software professionals to have a portfolio? I would guess that companies are the main driving force behind not having portfolios that can be shown, but at the same time they are thereby hurting themselves because they can't properly evaluate employees.