Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Three hundred programming interviews in thirty days

blog.triplebyte.com

Post reply on HN
Print view View original

91–100 of 248 posts

Re: Three hundred programming interviews in thirty days

#91

Post by foobarqux »

overgryphon wrote: post #77
ammon wrote: post #27

Earlier quoted context omitted.

Yeah, we agree. We don't plan on asking everyone to do hours of work on their own. A lot of people don't have the time. However, we're also seeing a bunch of applicants who are good programmers, but become really stressed in interviews and do badly (they freeze). These people have trouble getting jobs. What we want to do is offer them the option of doing a larger project on their own, rather than our final interview.…

Interview environments are always stressful, but so are other common workplace situations. Being unable to manage stress effectively might contribute to poor on-the-job performance, even for people with great pure coding ability. The process right now seems focused on finding people who are the best at only coding. In the future, do you intend to also consider communication skills? Senior developers act as mentors fo…

Interview stress is of the type I call "defusing the bomb": everything at stake, essentially immediate time constraints and confrontational. This never occurs in real life.

Re: Three hundred programming interviews in thirty days

#92

Post by ammon »

overgryphon wrote: post #77
ammon wrote: post #27

Earlier quoted context omitted.

Yeah, we agree. We don't plan on asking everyone to do hours of work on their own. A lot of people don't have the time. However, we're also seeing a bunch of applicants who are good programmers, but become really stressed in interviews and do badly (they freeze). These people have trouble getting jobs. What we want to do is offer them the option of doing a larger project on their own, rather than our final interview.…

Interview environments are always stressful, but so are other common workplace situations. Being unable to manage stress effectively might contribute to poor on-the-job performance, even for people with great pure coding ability. The process right now seems focused on finding people who are the best at only coding. In the future, do you intend to also consider communication skills? Senior developers act as mentors fo…

I think it's really on the interviewers (and us) to show that people who freeze in tech interviews would also do poorly in a job. It's probably sometimes true, but many times not.

Re: Three hundred programming interviews in thirty days

#93

Post by jackieDQ »

jrs235 wrote: post #60
techiferous wrote: post #52

Earlier quoted context omitted.

I've had a different experience on the hiring side of the table. The coding exercise that we provide as part of the interview has been very successful in surfacing the skill and talent of the programmer. By using that screening device, we have assembled a team of very sharp programmers. It's totally worth filtering out people who choose not to spend the time. Even though we end up losing some potential candidates, it…

We're currently hiring and for the first time, we should have started doing this years ago, given a simple coding exercise. I estimate it would/should take any programmer/problem solver no more than 15, maybe 20, minutes to come up with the solution if they have experience in the language we request they solve it in. The exercise doesn't require specific domain knowledge but should quickly assess whether a programmer…

See a simple one like that where if I need to spend time researching the question will only take up to an hour is one thing. But the last one I got from one of the video game engine companies required building a full crud app with live db, documented api, and multiple feature sets. all for a front end job. Some places just take it too far and sour the idea.

Re: Three hundred programming interviews in thirty days

#94

Post by KingMob »

ammon wrote: post #68
KingMob wrote: post #55

Earlier quoted context omitted.

I appreciate that you guys might not be statisticians, but if you're going to try and analyze data like this, you simply must address survival bias. As it stands, these data are meaningless unless you assume dropouts are completely unrelated to your screening. You claim doing well on the Fizzbuzz wasn't correlated with interview performance, but you also said "We saw twice the drop off rate on the coding problems as…

All candidates did both screens (quiz and fizzbuzz). The correlations were calculated against the same population. Now, I agree that survivor bias could affect the quality of these results (we know nothing about the significant % of people who dropped out). But it's not really possible to solve that problem outside of a lab. I don't think it's an argument to not do analysis. For now we're simply trying to minimize th…

Well, the candidates doing both screens is better, but it doesn't totally solve your problems.

It doesn't address the survival bias issue, and when you say a significant percentage dropped out, that's not reassuring. But it's not the case that you need a lab to solve the problem. Even a basic questionnaire of programming ability self-assessment might tell you if there are meaningful differences in the population that quits your process. At the very least, you should understand and talk about survival bias in your article to indicate you're aware of the issue.

Even if you still want to claim a difference between the quiz and coding exercise, you're not yet in the clear. For example, did you counterbalance the order you gave them to people? E.g., if everybody did the quiz first and the fizzbuzz second, that meant they were mentally fresher for the quiz and slightly more tired for the fizzbuzz, which could again create a spurious result. And this definitely doesn't require a lab to test.

Don't misunderstand me, I appreciate your attempts to quantify all this, and I actually think you guys have roughly the correct result (given the limited nature of fizzbuzz-style coding), but when you step into the experimental psych arena, you need to learn how to properly analyze your data. Given that your business is predicated on analyzing the results of how your hires do in the real world, you need to really up your analytical game.

Re: Three hundred programming interviews in thirty days

#95

Post by ammon »

dagw wrote: post #30
ammon wrote: post #15

Earlier quoted context omitted.

Hey! Author here. Yes, we did not know that the screening steps were meaningful, so for the first 300 applicants, we interviewed everyone, even people who performed badly. We then looked for correlation between screening step scores, and programming interview results. Doing well on the fizzbuzz problems was not very correlated. By dropoff I mean people who left during a step, and never logged back into our site.

Doing well on the fizzbuzz problems was not very correlated. I think you're mis-using fizbuzz. You cannot really 'do well' on it. You can basically pass it or fail it. Failing means you're probably no good, but passing it doesn't prove anything.

Yeah, I just mean "short programming problems of a difficultly similar to fizzbuzz"

Re: Three hundred programming interviews in thirty days

#96

Post by tokenadult »

The blog post kindly shared here reports on a startup's experiments with offering a new kind of hiring screening service. "We launched Triplebyte one month ago, with the goal of improving the way programmers are hired. . . . Well, a little over a month has now passed. In the last 30 days, we've done 300 interviews. We've started to put our ideas into practice, to see what works and what doesn't, and to iterate on our process." They are currently validating what they are doing on the first steps just against what happens at the later steps in their process: "For now, we're evaluating all of our experiments against our final round interview decisions. This does create some danger of circular reasoning (perhaps we're just carefully describing our own biases)."

I agree with the blog post author that current hiring processes mostly show that "too many companies are content to do what they've always done." And the idea of a standardized, automated quiz of programming knowledge sounds interesting. But what has to happen next is to an actual validation study and find out if programmers hired by this process do better as programmers in actual workplaces than programmers hired by some other process.

Regular readers of HN are aware that I have a FAQ post about this topic of company hiring procedures.[1] Company hiring procedures are the research focus of industrial and organizational psychologists, who have almost a century of research to look back on with long-term, large n studies to provide data on what works and what doesn't work for hiring capable workers. It's a shame that most company human resource departments ignore basically all of that research.

[1] https://news.ycombinator.com/item?id=4613543

Re: Three hundred programming interviews in thirty days

#97

Post by orkoden »

cogg wrote: post #75
rezistik wrote: post #33

Earlier quoted context omitted.

I personally prefer it to a point. My only request when I get this type of interview is that I'm allowed to make the project and requirements public on my github for future interviewers. If I'm doing a free project I feel it's fair.

I am curious why you have that requirement. What is the fairness you are striving for?

If I invest time to do something, it better be worth it for me.

Re: Three hundred programming interviews in thirty days

#98

Post by ruswick »

protonfish wrote: post #6

It's great to see some objective research being done on this, and I am very interested in following the results. They mention evaluating the effectiveness of giving a candidate a project to do "in their own time." I recently had a interview that included this and I can share the result: I accepted an offer from a different company that didn't require it. I doubt my life is that different than anyone else's, with a fu…

Also keep in mind that applying to jobs is frequently throwing your application into a black hole and just praying. Sending a resume into the black hole is one thing. Sending a large codebase or project you worked on as part of the application is entirely different.

I remember a few months ago, I was applying for an internship with a company and they requested that I write a web app to test coding proficiency before they thought about granting an interview. I spent about 12-15 hours writing the thing, deployed the app to Heroku and put the code on Github. I sent a link to the app and the repo to the recruiter. I never got a response of any kind.

Although some companies do try to make the application experience as pleasant as possible for the applicant, the majority of companies don't give a shit about the people who want to work for them, and make no attempt to respect the time of their applicants. Placing greater demands on the applicant is an easy way to shift more of the workload from the recruiter to the applicant. The applicant is the loser in this situation.

Re: Three hundred programming interviews in thirty days

#99

Post by dfan »

protonfish wrote: post #6

It's great to see some objective research being done on this, and I am very interested in following the results. They mention evaluating the effectiveness of giving a candidate a project to do "in their own time." I recently had a interview that included this and I can share the result: I accepted an offer from a different company that didn't require it. I doubt my life is that different than anyone else's, with a fu…

My last company had a screening coding problem that was designed to take around a day (although it didn't have to; when I did it as a reference point, it took a couple of hours). I was always kind of astonished how many applicants were willing to do the whole thing, given, as you say, that they often already had a full-time job and/or were applying for multiple positions.

We always gave really extensive feedback on it, partially because one of the things were were looking for was how people would react to that feedback (I was generally surprised at how many people got defensive about fairly objective observations/questions).

Overall I think it was quite a good data point. It was nice to have a more low-key and extensive complement to the more stressful live whiteboard interview (yes, we did have one of those, and I still think it was useful).

Re: Three hundred programming interviews in thirty days

#100

Post by ConfuciusSay »

jbangert wrote: post #44

Very interesting methodology, but it would be very nice to correlate this data with long-term job performance. Interview decisions (of which of course you get more than long-term results, and they are clearer to quantify) are hopefully, but not necessarily an indicator of whether an employee works out for your company. Otherwise you run the risk of optimizing the quiz/screening process around metrics that influence y…

This is exactly what I came here to say.

It's all well and good that certain things (quiz) correlate more to performing well in a long-form remote coding task.

But part of the problem of hiring is that these test-style code questions are themselves not necessarily good indicators of future job performance.

The only way to truly analyze whether a given technique is working or not is to follow the candidates through into their jobs and see which ones actually become good long term employees.

Post reply on HN