The chart is kind of dumb IMHO. It's nice that they show correlation between different factors and their final hiring decisions, that doesn't indicate the actual quality of the people hired. All it does is reveal an unwritten formula behind their hiring decisions. In other words, the time to complete a programming task is being ignored in their decision process and so should be removed from the process. That's independent of how well it indicates a good candidate. Or am I missing something?
Three hundred programming interviews in thirty days
21–30 of 248 posts
Re: Three hundred programming interviews in thirty days
#22Correlation between testing and interviewing success tells you nothing about the correlation between either of them and actual work performance.
From the article, in bold: The really exciting point comes when we can re-run all this analysis, basing it on actual job performance, rather than interview results
How will they gather data about the job performance of the people who did not pass the interview? Without that, the study is only halfway done.
Re: Three hundred programming interviews in thirty days
#23Imo, this post did not organize its data and findings into a coherent presentation. For example... >The fizzbuzz-style coding problems, however, did not perform as well. While the confidence intervals are large, the current data shows less correlation with interview results. [...] The coding problems were also harder for people to finish. We saw twice the drop off rate on the coding problems as we saw on the quiz. I…
Hey! Author here. Yes, we did not know that the screening steps were meaningful, so for the first 300 applicants, we interviewed everyone, even people who performed badly. We then looked for correlation between screening step scores, and programming interview results. Doing well on the fizzbuzz problems was not very correlated. By dropoff I mean people who left during a step, and never logged back into our site.
>Doing well on the fizzbuzz problems was not very correlated.
If you mean "correlation" to only refer to the population that passed fizzbuzz, then it is to be expected that the final positive/accepted interview evaluations don't correlate. Fizzbuzz was never statistically designed for that. It was designed for early rejection and not for predicting ultimate success at the end of a multi-step interview cycle.
>By dropoff I mean people who left during a step, and never logged back into our site.
And the population mentioned in this sentence is what I first interpreted to be included in your "non-correlation". It looks like you don't include this population. The quitters that never logged back in were not further tested by you for later stage evaluations. That's where my confusion was and now it's resolved.
Re: Three hundred programming interviews in thirty days
#24"as soon as we started doing them however, I saw a problem. Almost everyone was passing."
Why is that a problem? Maybe almost everyone is decently good (as evidenced by having a string of jobs, and presumably, references), and your interviews are creating tons of false negatives.Or heck, vice versa. You don't know.
You are presuming your conclusions. You have no basis to make conclusions yet, you just have incomplete data. It's iteresting data, and I'm gleefully happy that somebody is looking at this in the context of programmers (too many studies are very broad, across many different career/job type, IMO). But I think all you have right now is data. Fishing for correlations at this point is nearly bound to lead you astray.
With that aside, I'm very interested in the eventual correlation with test performance and job performance. I'm biased - I dislike IQ tests, but I must admit there is a lot of research on them out there. For me personally, I perform spectacularly on this sort of test, pretty poorly in whiteboard tests, so-so in pair program to get a job, and generally top of the heap in actual job performance. It would definitely help me personally if these tests were true. Yet, still, I wonder, do they measure "get things done"? Do they measure "don't piss off the CEO/customer" skills? There's a ton of things that I think are important beyond pure cognitive skills.
Re: Three hundred programming interviews in thirty days
#25I interviewed with TripleByte and it was different from a normal job interview in some ways, but fundamentally also the same. There is just no culture fit aspect to the interview. Keep in mind that they are not addressing the problem that coding interviews do not predict on-the-job success. They are basically addressing the needs of a broken system.
Re: Three hundred programming interviews in thirty days
#26This is interesting, but are they using "performance on a sample coding task" as the truth? If we knew that was a good proxy for being a good hire, we'd already be done.
Re: Three hundred programming interviews in thirty days
#27It's great to see some objective research being done on this, and I am very interested in following the results. They mention evaluating the effectiveness of giving a candidate a project to do "in their own time." I recently had a interview that included this and I can share the result: I accepted an offer from a different company that didn't require it. I doubt my life is that different than anyone else's, with a fu…
Yeah, we agree. We don't plan on asking everyone to do hours of work on their own. A lot of people don't have the time. However, we're also seeing a bunch of applicants who are good programmers, but become really stressed in interviews and do badly (they freeze). These people have trouble getting jobs. What we want to do is offer them the option of doing a larger project on their own, rather than our final interview. I think this might help a lot of people (clearly we have to test this)
Re: Three hundred programming interviews in thirty days
#28>Suddenly, a significant percentage of the people who had spoken well about impressive-sounding projects failed, in some cases spectacularly, when given relatively simple programming tasks.
I feel like this should be listed among the least surprising things in the world. Being a good programmer is about MAKING that hard project look easy, by approaching it in the correct way!
Re: Three hundred programming interviews in thirty days
#29I got a phone interview with Harj and wasn't considered for anything further.
I'm not sure what kind of hackers they were looking for, but I've been directly involved with creating the infrastructure used in marketing campaigns with the likes of CNN, McDonalds, Infiniti, and more. I've turned an idea into a company with 8 full time employees and have investors seriously interested in one of my side projects. I'm currently involved with leading a project that integrates with a large bank.
I'm a full stack ruby dev learning clojure in my spare time and heavily involved with self improvement. Anyone who watches me for a moment can see that I can solve problems very quickly. I didn't care much about being selected, I have a solid job and offers coming in.
Would anyone who got selected by Triplebyte care to list their credentials/achievements? My main motivation was to see how I compare against others at my current level.
Re: Three hundred programming interviews in thirty days
#30Imo, this post did not organize its data and findings into a coherent presentation. For example... >The fizzbuzz-style coding problems, however, did not perform as well. While the confidence intervals are large, the current data shows less correlation with interview results. [...] The coding problems were also harder for people to finish. We saw twice the drop off rate on the coding problems as we saw on the quiz. I…
Hey! Author here. Yes, we did not know that the screening steps were meaningful, so for the first 300 applicants, we interviewed everyone, even people who performed badly. We then looked for correlation between screening step scores, and programming interview results. Doing well on the fizzbuzz problems was not very correlated. By dropoff I mean people who left during a step, and never logged back into our site.
Doing well on the fizzbuzz problems was not very correlated.
I think you're mis-using fizbuzz. You cannot really 'do well' on it. You can basically pass it or fail it. Failing means you're probably no good, but passing it doesn't prove anything.