Live data from Hacker News

Three hundred programming interviews in thirty days

blog.triplebyte.com

51–60 of 248 posts

Re: Three hundred programming interviews in thirty days

#51
post #38

It's great to see some objective research being done on this, and I am very interested in following the results. They mention evaluating the effectiveness of giving a candidate a project to do "in their own time." I recently had a interview that included this and I can share the result: I accepted an offer from a different company that didn't require it. I doubt my life is that different than anyone else's, with a fu…

I'm kind of in a similar boat. I've done them for companies that I like in the past, however I flat out refuse to do them for recruiting / job placement companies. ammon / uptown, I've put off doing my screen-sharing section of your interview process (I got pushed past the phone screen?) due to a lack of time. Maybe adding a way to 'schedule' it would be helpful for those of us that live by our g.calendars....and you…

What would be the best experience for you to schedule the interview? We'd definitely like to make that easier.

Re: Three hundred programming interviews in thirty days

#52

It's great to see some objective research being done on this, and I am very interested in following the results. They mention evaluating the effectiveness of giving a candidate a project to do "in their own time." I recently had a interview that included this and I can share the result: I accepted an offer from a different company that didn't require it. I doubt my life is that different than anyone else's, with a fu…

I've had a different experience on the hiring side of the table. The coding exercise that we provide as part of the interview has been very successful in surfacing the skill and talent of the programmer. By using that screening device, we have assembled a team of very sharp programmers. It's totally worth filtering out people who choose not to spend the time. Even though we end up losing some potential candidates, it's worth it for a 0% chance of a mishire for technical reasons.

Re: Three hundred programming interviews in thirty days

#53
OPEN LETTER ================

Hi, I had a reading of the conclusions you made and I felt as if you defined a process of hiring machines to code rather than humans. So I took a few moments to read your manifesto(https://triplebyte.com/manifesto) (the premise on which your entire conclusion is made), and here is my take on it.

1. /"Whiteboard coding and algorithm questions aren't good predictors of how effective someone will be at writing real code." Whiteboard coding show how someone really thinks. It illustrates the though process of the person and that helps the interviewer to judge him on his rational thinking and his logical approaches. Algorithms add to this by illustrating the problem solving ability. A person may not be able to solve an Algorithm actually, but the attempt on a whiteboards speaks more than his implementation on a online platform.

2./ "Candidates deserve a consistent experience and consistent evaluation". The entire USP of an interview is the diversity which allows the interviewer to judge if someone is able to adapt to new situations and come out of his comfort zone. What you are suggesting is to change the interview process into a GRE exam which will only in-turn develop the culture among developers to prepare for that exam for 2 years.

3./"Hiring decisions should be made using a clear scoring system, not gut feelings" Most of the companies have a 3round or 4 round interview process. It is obvious enough to remove the gut feeling factor. If you wanna argue that it may be possible that a candidate got selected based on the gut feeling of all 4 interviewers then my counter argument is that he is worth being selected if he could generate that gut feeling in so many people.

4./"The hiring process should be focused on discovering strengths, not uncovering weaknesses" Agree to the point. However, the irony is that you are trying to define a particular process to hiring. I wonder if it could actually perform the "discovery" part.

5./"Candidates should be told exactly what to expect in an interview, and be allowed to prepare in advance. " So basically you want to hire the guy who has studied the most over the smartest guy in the room. From my experience, I can surely say if companies like "Google" and "Fb" used to follow that practice, I wouldn't be even writing their name here.

6./"truthful feedback on how they did so they know how they can improve" Agreed. Something that should be adapted by all companies in their recruiting process.

7./"Good programmers come from all types of background." You enforce my point with this statement. Good programmers need not be just people who could quickly write a program for a search in a large of data using hash maps but can also be people who have brilliant problem solving ability and are slow in transforming that into code, or people who are amazing in thinking software design and scalability and probably cannot remember code syntax so well. A company needs a good blend of all these people. Then only a good ecosystem to growth is created rather than just making a team of 10 machines who could transform a pseudo code into Java in 10 minutes.

8./" The software industry needs to experiment more with hiring processes and figure out what really works." I think many are already doing that by doing Tech Hackathons, online challenges, weekend projects, open source contribution references etc. So, not something new which you guys figured out.

9./"Candidates are at a fundamental disadvantage in salary and equity negotiations" Not sure what kind of companies you have surveyed. I think most well known companies maintain clear standards of salary and compensation plan. Though people will surely be flattered reading this. :)

10./"Companies should not have to make recruiting a core competency" Now you are just trying to open the market for yourself by yourself. No comments. :P

Would love to hear your counter arguments. Mail me. :-)

Re: Three hundred programming interviews in thirty days

#54
post #18

Here's the part that really seems to matter the most. This does create some danger of circular reasoning (perhaps we're just carefully describing our own biases). But we have to start somewhere, and basing our evaluations on how people write actual code seems like a good place.The really exciting point comes when we can re-run all this analysis, basing it on actual job performance, rather than interview results. Abso…

Author here. We do do the interviews blind to earlier screening results. That's clearly vital. But you're totally right that job performance is the real thing that matters

So, the jury is out until a followup. How to measure productivity without bias? Applying Demarco and Lister's "Coding War Games" approach may work between companies, but how to apply within a single company? What would be the standard error? A dozen new hires is a lot different than 300 interviewees.

What kind of skill set? My previous company contracted with Galye Laakmann McDowell to coach the engineers before an acquisition due diligence. I thought it was unnecessary because we rarely used O() notation in our day-to-day. Since then, I'm designing a machine learning product and find sections in McDowell's book extremely relevant. My takeaway is that it all depends: are you hiring an architect or a hiring a construction crew? How do you measure such different skill sets? Or, how to measure with a job that combines both?

Re: Three hundred programming interviews in thirty days

#55
post #15
post #4

Imo, this post did not organize its data and findings into a coherent presentation. For example... >The fizzbuzz-style coding problems, however, did not perform as well. While the confidence intervals are large, the current data shows less correlation with interview results. [...] The coding problems were also harder for people to finish. We saw twice the drop off rate on the coding problems as we saw on the quiz. I…

Hey! Author here. Yes, we did not know that the screening steps were meaningful, so for the first 300 applicants, we interviewed everyone, even people who performed badly. We then looked for correlation between screening step scores, and programming interview results. Doing well on the fizzbuzz problems was not very correlated. By dropoff I mean people who left during a step, and never logged back into our site.

I appreciate that you guys might not be statisticians, but if you're going to try and analyze data like this, you simply must address survival bias. As it stands, these data are meaningless unless you assume dropouts are completely unrelated to your screening.

You claim doing well on the Fizzbuzz wasn't correlated with interview performance, but you also said "We saw twice the drop off rate on the coding problems as we saw on the quiz."

An alternate explanation for your finding then is, more low-quality candidates drop out of the process when given FizzBuzz, leaving a relatively homogeneous pool of higher-quality candidates for the later interview. This effectively reduces the ratio of meaningful interindividual differences relative to noise, which will reduce the correlations.

In all likelihood, both of your correlations are low, but the idea that coding is less predictive than a quiz could be purely a statistical fluke due to survivor bias.

Re: Three hundred programming interviews in thirty days

#56

It's great to see some objective research being done on this, and I am very interested in following the results. They mention evaluating the effectiveness of giving a candidate a project to do "in their own time." I recently had a interview that included this and I can share the result: I accepted an offer from a different company that didn't require it. I doubt my life is that different than anyone else's, with a fu…

My rule for these - only do these tests in-person, on-site, and after after the company has invested a significant amount of their employees' time to interview you in-person.

Tons of companies hand out these "homework" tests to hundreds of people without having even the slightest intention of hiring them. F* that.

Re: Three hundred programming interviews in thirty days

#57
post #45
post #27

Earlier quoted context omitted.

Yeah, we agree. We don't plan on asking everyone to do hours of work on their own. A lot of people don't have the time. However, we're also seeing a bunch of applicants who are good programmers, but become really stressed in interviews and do badly (they freeze). These people have trouble getting jobs. What we want to do is offer them the option of doing a larger project on their own, rather than our final interview.…

I had an enormous amount of trouble interviewing / whiteboarding initially but I got used to it pretty fast. However, I would still prefer a take home test as I feel its a far better qualifier than an in-person interview.

A lot of people feel this way, and I see no reason why it can't be an option at more companies

Re: Three hundred programming interviews in thirty days

#58

It's great to see some objective research being done on this, and I am very interested in following the results. They mention evaluating the effectiveness of giving a candidate a project to do "in their own time." I recently had a interview that included this and I can share the result: I accepted an offer from a different company that didn't require it. I doubt my life is that different than anyone else's, with a fu…

I've had a different experience on the hiring side of the table. The coding exercise that we provide as part of the interview has been very successful in surfacing the skill and talent of the programmer. By using that screening device, we have assembled a team of very sharp programmers. It's totally worth filtering out people who choose not to spend the time. Even though we end up losing some potential candidates, it…

A take-home test has a 0% chance of being cheated on? Are you sure you're talking about the same thing here?

Re: Three hundred programming interviews in thirty days

#59
post #51
post #38

Earlier quoted context omitted.

I'm kind of in a similar boat. I've done them for companies that I like in the past, however I flat out refuse to do them for recruiting / job placement companies. ammon / uptown, I've put off doing my screen-sharing section of your interview process (I got pushed past the phone screen?) due to a lack of time. Maybe adding a way to 'schedule' it would be helpful for those of us that live by our g.calendars....and you…

What would be the best experience for you to schedule the interview? We'd definitely like to make that easier.

Just something simple, allow me to add it to my google calendar.

Re: Three hundred programming interviews in thirty days

#60

It's great to see some objective research being done on this, and I am very interested in following the results. They mention evaluating the effectiveness of giving a candidate a project to do "in their own time." I recently had a interview that included this and I can share the result: I accepted an offer from a different company that didn't require it. I doubt my life is that different than anyone else's, with a fu…

I've had a different experience on the hiring side of the table. The coding exercise that we provide as part of the interview has been very successful in surfacing the skill and talent of the programmer. By using that screening device, we have assembled a team of very sharp programmers. It's totally worth filtering out people who choose not to spend the time. Even though we end up losing some potential candidates, it…

We're currently hiring and for the first time, we should have started doing this years ago, given a simple coding exercise.

I estimate it would/should take any programmer/problem solver no more than 15, maybe 20, minutes to come up with the solution if they have experience in the language we request they solve it in. The exercise doesn't require specific domain knowledge but should quickly assess whether a programmer, particularly with experience in a different language, is able to learn and apply the syntax and semantics of another language. If they are completely unfamiliar with the language then maybe an hour. They can use whatever references they want. We will check for complete copy/pasted code and alter the requirements and/or test data over time.

The other great benefit is we get to see what/if/how they ask questions to clarify the problem and solution.

EDIT: This exercise is geared towards fresh graduates or developers that don't have experience in the language we use. Can it be gamed? Yes. Might some people not bother? Yes. Are we okay with that? Yes.

Post reply on HN