Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Terrible Technical Interview

techcrunch.com

Post reply on HN
Print view View original

231–236 of 236 posts

Re: The Terrible Technical Interview

#231

Post by sytelus »

fsk wrote: post #228
sytelus wrote: post #223

Earlier quoted context omitted.

No, I did not say I hire 10% of candidates I interview :). What I said was that's a fair estimate in industry. I feel you are truly confused about FP and FN. Whether there are 99% bad developers out there or if you hire 10% of candidates you interview - these both quantities are independent of FN and FP. FN says that you are turning away X good people and it's again independent of FP which ultimately decides how many…

If you increase FN and FP stays the same, you are more likely to make a bad hire. Do you understand this?

May be I'm completely missing something here but my understanding is this: FP = (good hires you made) / (all hires you made). Your likelyhood of making bad hire is 1-FP. If you hired 100 people and your FP was 10% then on average you would have 10 bad hires on your team. So FP determines the number of bad hires you would eventually have. FN has nothing to do with it - it only determines how long before you make a good hire, it doesn't influence actual number of bad hires you will make.

I'm using standard terminologies here. There are plenty of textbooks and articles on confusion matrix, precision, recall, RoC etc. Not sure what definitions you are using to arrive at conclusion that FN increases the number of good hires (it only increases effort).

Re: The Terrible Technical Interview

#232

Post by fsk »

sytelus wrote: post #231
fsk wrote: post #228

Earlier quoted context omitted.

If you increase FN and FP stays the same, you are more likely to make a bad hire. Do you understand this?

May be I'm completely missing something here but my understanding is this: FP = (good hires you made) / (all hires you made). Your likelyhood of making bad hire is 1-FP. If you hired 100 people and your FP was 10% then on average you would have 10 bad hires on your team. So FP determines the number of bad hires you would eventually have. FN has nothing to do with it - it only determines how long before you make a goo…

You aren't using statisics correctly.

FP = probability, given a bad candidate, you will hire him

FN = probability, given a good candidate, you will pass

Suppose 100 bad candidates, 10 good candidates

FP=10%, FN=10%

You make 10 bad hires and 9 good hires

FP=10%, FN=20%

You make 10 bad hires and 8 good hires.

So increasing FN lowers your yield.

Your statstic (good hires / total hires) tells you nothing about your actual FP or FN value.

If you don't get it, I'm not wasting time on you anymore. You are very dangerous. You think you know statistics, but you don't.

Re: The Terrible Technical Interview

#233

Post by fsk »

fsk wrote: post #232
sytelus wrote: post #231

Earlier quoted context omitted.

May be I'm completely missing something here but my understanding is this: FP = (good hires you made) / (all hires you made). Your likelyhood of making bad hire is 1-FP. If you hired 100 people and your FP was 10% then on average you would have 10 bad hires on your team. So FP determines the number of bad hires you would eventually have. FN has nothing to do with it - it only determines how long before you make a goo…

You aren't using statisics correctly. FP = probability, given a bad candidate, you will hire him FN = probability, given a good candidate, you will pass Suppose 100 bad candidates, 10 good candidates FP=10%, FN=10% You make 10 bad hires and 9 good hires FP=10%, FN=20% You make 10 bad hires and 8 good hires. So increasing FN lowers your yield. Your statstic (good hires / total hires) tells you nothing about your actua…

Your statistic (good hires / total hires), using the jargon from your link, is precision or positive predictive value (PPV).

Using the math from that link, if you decrease FN, then PPV increases.

Re: The Terrible Technical Interview

#234

Post by [deleted] »

fsk wrote: post #233
fsk wrote: post #232

Earlier quoted context omitted.

You aren't using statisics correctly. FP = probability, given a bad candidate, you will hire him FN = probability, given a good candidate, you will pass Suppose 100 bad candidates, 10 good candidates FP=10%, FN=10% You make 10 bad hires and 9 good hires FP=10%, FN=20% You make 10 bad hires and 8 good hires. So increasing FN lowers your yield. Your statstic (good hires / total hires) tells you nothing about your actua…

Your statistic (good hires / total hires), using the jargon from your link, is precision or positive predictive value (PPV). Using the math from that link, if you decrease FN, then PPV increases.

[deleted]

Re: The Terrible Technical Interview

#235

Post by sytelus »

fsk wrote: post #233
fsk wrote: post #232

Earlier quoted context omitted.

You aren't using statisics correctly. FP = probability, given a bad candidate, you will hire him FN = probability, given a good candidate, you will pass Suppose 100 bad candidates, 10 good candidates FP=10%, FN=10% You make 10 bad hires and 9 good hires FP=10%, FN=20% You make 10 bad hires and 8 good hires. So increasing FN lowers your yield. Your statstic (good hires / total hires) tells you nothing about your actua…

Your statistic (good hires / total hires), using the jargon from your link, is precision or positive predictive value (PPV). Using the math from that link, if you decrease FN, then PPV increases.

Sorry, I did mixed up precision in my reply. I just got time to think about this whole debate more carefully and I realize you are actually right if we fix up some of the terminology you have used. The mis-statements and confusion on my part has occurred due to this terminology differences.

First FP and FN are not probabilities. They are just unbounded numbers. This may feel pedantic but in a moment I'll show you why this is critical. Let me draw the confusion matrix first (G = Good candidates, H = Hired candidate etc):

\ H NH \--------- G | TP FN B | FP TN

What you are referring to as probabilities is actually False Positive Rate or FPR and TNR respectively which is defined as follows:

  FPR = FP / (FP + TN) = FP/B
  FNR = FN / (TP + FN) = FN/G
Now the quantity you are after is probability that given you did hiring and ended up with good guy which is, nothing but precision:

precision = P(G|H) = TP/H

So how do we get TP to calculate precision if we only knew FPR, FNR, G and B? I did little equation gymnastics using above and got below:

TP = G - GFNR H = TP + FP = TP + FPRB

So now you can plug this in to above equation for precision and find that as you increase FNR, precision goes down while you keep FPR constant. So you are actually correct. Although it might look like unnecessary exercise vs following intuition I think above equation can actually help calculate exact drop in precision and multiply that with cost of FP vs FN to get the operating sweet spot. On my part I need to do some soul searching to figure out why this didn't triggered to me before :).

Re: The Terrible Technical Interview

#236

Post by jerematasno »

politician wrote: post #218
tptacek wrote: post #201

Earlier quoted context omitted.

I've carefully read every comment on this subthread, and here's what I think: This question is a trifecta of ineffective candidate screening tactics: (a) It's a technical screening question, one a strong candidate could get wrong, based on a technical aptitude that is trivial to teach on the job and thus rarely worth paying a premium for. (b) It's a subjective technical question, for which reasonable engineers can ha…

Thanks for being open about your perspective on the interviewing process, and for taking the time to follow up on these threads. Do you believe that technical phone pre-screens are ineffective in general? From what I've read Matasano doesn't pre-screen candidates, but provides complementary study materials instead. Is that because the subject matter is specialized? Would you approach hiring web dev roles differently?…

In general, my feeling is that the Matasano process (which I currently manage) works outstandingly well where there isn't a flood of qualified candidates. If you have a glut of folks who are ready to start working, you can get away with a terrible process.

We do free-form technical interviews, but only to try to detect candidates who really aren't ready for the work-sample challenges. Our in-person interviews are standardized and try to evaluate consulting/architecture skills that are hard (impossible?) to measure without people. These involve open-ended intermediary questions, but the final answers are structured.

The bottom line is this: you cannot compare candidates using free-form interviews. You must compare candidates who are going to be doing similar work. Thus, free-form interviews have no evaluative value.

Post reply on HN