Three hundred programming interviews in thirty days • Hacker News Board

Re: Three hundred programming interviews in thirty days

#81

Post by etrautmann » Tue, Jun 23, 2015, 8:32 PM UTC

"The really exciting point comes when we can re-run all this analysis, basing it on actual job performance, rather than interview results"

I'm not sure how this gets around the circularity arguments though, since you never get to evaluate the job performance of someone you selected out already. Only the tiny fraction of coders that make it past the initial test get evaluated, which could serve to reinforce the potential biases rather than ameliorate them.

The one case in which this would work is if they hired a number of coders that didn't work out well, and could add or update a feature as a negative predictor of job success.

I'm assuming that they're not at the scale of a larger company with thousands of engineers, and that the observations going into a regression model are relatively sparse. If this is a startup with a 20 hires, I'd be surprised if there was much to do to refine the model after a round or two of evaluations, but would be excited to learn otherwise.

Re: Three hundred programming interviews in thirty days

#82

Post by vonmoltke » Tue, Jun 23, 2015, 8:32 PM UTC

overgryphon wrote: post #77

ammon wrote: post #27

Earlier quoted context omitted.

Yeah, we agree. We don't plan on asking everyone to do hours of work on their own. A lot of people don't have the time. However, we're also seeing a bunch of applicants who are good programmers, but become really stressed in interviews and do badly (they freeze). These people have trouble getting jobs. What we want to do is offer them the option of doing a larger project on their own, rather than our final interview.…

Interview environments are always stressful, but so are other common workplace situations. Being unable to manage stress effectively might contribute to poor on-the-job performance, even for people with great pure coding ability. The process right now seems focused on finding people who are the best at only coding. In the future, do you intend to also consider communication skills? Senior developers act as mentors fo…

"Stress" is not some monolithic, universal feeling. This line gets trotted out frequently in these discussions, but no one has shown a correlation between the types of stress induced in interview situations and the types of stress induced in whatever situations are typical for your company. I would hesitate to even equate the stress of high-pressure situations in different companies.

Re: Three hundred programming interviews in thirty days

#83

Post by etrautmann » Tue, Jun 23, 2015, 8:32 PM UTC

"The really exciting point comes when we can re-run all this analysis, basing it on actual job performance, rather than interview results"

I'm not sure how this gets around the circularity arguments though, since you never get to evaluate the job performance of someone you selected out already. Only the tiny fraction of coders that make it past the initial test get evaluated, which could serve to reinforce the potential biases rather than ameliorate them.

The one case in which this would work is if they hired a number of coders that didn't work out well, and could add or update a feature as a negative predictor of job success.

I'm assuming that they're not at the scale of a larger company with thousands of engineers, and that the observations going into a regression model are relatively sparse. If this is a startup with a 20 hires, I'd be surprised if there was much to do to refine the model after a round or two of evaluations, but would be excited to learn otherwise.

Re: Three hundred programming interviews in thirty days

#84

Post by vonmoltke » Tue, Jun 23, 2015, 8:38 PM UTC

michaelvkpdx wrote: post #72

Most of the problems of coding interviews can be solved by good reference checking. Code problems in isolation will never expose how a developer will do when faced with real-world issues, users, QA, and teammates. Why are people so afraid to pick up the phone and talk to references? I'm always happy to give out my references, and always delighted to talk about the good devs I've worked with, with specifics about what…

> Why are people so afraid to pick up the phone and talk to references?

First, references are chosen by the candidate, so they are almost universally expected to say good things.

Second, many companies explicitly prohibit their employees from giving meaningful references about former employees. Relying on a process that needs people to violate company policies or to have moved on from the role wherein they worked with the candidate is not really a good idea.

Re: Three hundred programming interviews in thirty days

#85

Post by dikaiosune » Tue, Jun 23, 2015, 8:39 PM UTC

Neat idea, especially for someone like me who's trying to get out of support and into development.

From Triplebytes' FAQ:

"When do I meet companies?

If we decided to work together after giving you feedback post our technical interviews, we'll start introducing you to the companies and guiding you through their hiring process."

So, just to be clear, first you quiz/screenshare/interview with Triplebyte, and then you still have to go through each company's search process? Or do companies partner with Triplebyte to fast-track candidates who've already been vetted?

Re: Three hundred programming interviews in thirty days

#86

Post by solve » Tue, Jun 23, 2015, 8:39 PM UTC

techiferous wrote: post #71

solve wrote: post #58

Earlier quoted context omitted.

A take-home test has a 0% chance of being cheated on? Are you sure you're talking about the same thing here?

Our exercise can't be cheated on because we discuss it with the candidate during the in-person interview and even ask them to change the code in a simple way. Also, the exercise is designed to be original and so a solution can't be Googled.

We're talking only about the "in their own time" take-home types of tests. The type of cheating I'm talking about is outsourcing the problem or even just having a friend do it for them.

Re: Three hundred programming interviews in thirty days

#87

Post by jackieDQ » Tue, Jun 23, 2015, 8:39 PM UTC

geebee wrote: post #39

protonfish wrote: post #6

It's great to see some objective research being done on this, and I am very interested in following the results. They mention evaluating the effectiveness of giving a candidate a project to do "in their own time." I recently had a interview that included this and I can share the result: I accepted an offer from a different company that didn't require it. I doubt my life is that different than anyone else's, with a fu…

I did this once. After a take home quiz that took about an hour, I had a 6-7 hour "homework" problem. I did it, and sent it in. It took the company a month to get back to me, though I did stay in contact with the recruiter. The final answer, delivered by the recruiter, was "we've decided not to more forward at this time." No other feedback. I have no idea if anyone even looked at it - I never did speak to a dev at th…

Yep I've been burned a couple times by this and only under certain circumstances will I do a take home project. Even worse is when they also ask before hand not to post it to your github account. If they aren't going to give me feedback, then I'm going to host it and seek it out elsewhere.

Re: Three hundred programming interviews in thirty days

#88

Post by saidajigumi » Tue, Jun 23, 2015, 8:40 PM UTC

technofiend wrote: post #64

eitally wrote: post #16

Earlier quoted context omitted.

My wife is in this position now, and it's not for a programming job. She works in pharmaceuticals, in pharmacovigilance & risk management. She does a lot of data analysis & writing (the latest report she wrote for Health Canada & the FDA was 461 pages). A company she's interviewing with asked her to do a sample writing task as part of her application/interview. She hasn't decided whether it's worth it or not. For som…

I had a similar experience last year applying for a major technology company. Although I still lightly code in the devops sense (process automation) and to do typical management data analysis my day job is managing people, processes and crises. If you have billion dollar trading systems and need a cool-headed guy who can talk to VPs or directors in one breath and SAs, DBAs, SAN, Network, etc guys in the next I'm your…

> Am I wrong to think if I'm going to manage a 60-100 person organization across three timezones I don't really need to be under the hood in the code any longer?

An organization that does this is hiring stupidly. They have clearly demonstrated via interview that they don't understand the job to be done and/or how to determine that a candidate can do that job. Asking low-level technical questions is a terrible way to answer what I'll guess was the intended "question", something like:

Does this candidate understand the technology and work well enough to make executive decisions grounded in our organization's reality?

Re: Three hundred programming interviews in thirty days

#89

Post by vonmoltke » Tue, Jun 23, 2015, 8:40 PM UTC

brudgers wrote: post #78

gwbas1c wrote: post #63

Many fields require serious professional certification. You can't become a doctor unless you go through a board certification that includes simulated patient interaction. Likewise, you can't build a bridge or a dam without engineering certification. IMO, this article demonstrates the need to certify software engineers; using a process similar to the interviewing process described. Therefore, when hiring, we can skip…

Professional licencing processes establish minimum standards of competence, training and education. They do not guarantee excellence and a license represents that its holder is no worse than the worst legally allowable practitioner. Half of all surgeons are below average. You don't want one of them if you have a choice.

Depends on what the average and standard deviation are. If the average is shitty in the absolute sense, I probably want one well above average. If the average is stellar, I probably don;t care as long as I am not getting bottom of the barrel.

Re: Three hundred programming interviews in thirty days

#90

Post by contravariant » Tue, Jun 23, 2015, 8:41 PM UTC

jasode wrote: post #23

ammon wrote: post #15

Earlier quoted context omitted.

Hey! Author here. Yes, we did not know that the screening steps were meaningful, so for the first 300 applicants, we interviewed everyone, even people who performed badly. We then looked for correlation between screening step scores, and programming interview results. Doing well on the fizzbuzz problems was not very correlated. By dropoff I mean people who left during a step, and never logged back into our site.

>Doing well on the fizzbuzz problems was not very correlated. If you mean "correlation" to only refer to the population that passed fizzbuzz, then it is to be expected that the final positive/accepted interview evaluations don't correlate. Fizzbuzz was never statistically designed for that. It was designed for early rejection and not for predicting ultimate success at the end of a multi-step interview cycle. >By drop…

Agreed, it would be more useful to know the false negative rate (i.e. those incorrectly 'rejected' by fizzbuzz). The small / negative correlation could just be caused by a high number of false positives.