Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Harder programming questions do a worse job of predicting outcomes

triplebyte.com

Post reply on HN
Print view View original

101–110 of 540 posts

Re: Harder programming questions do a worse job of predicting outcomes

#101

Post by joker3 »

There's a problem here. The only thing that Triplebyte can claim based on their data is that easier programming questions are more predictive of performance among candidates who received an offer. Since candidates who get offers are (in theory) different from candidates who don't get offers, we can't necessarily generalize from one population to the other.

There's also a question about how to mix question difficulty. Should you ask nothing but easy questions, or is it good to throw in a harder question or two to see how the candidate reacts to something they can't answer? I can see a good interviewer getting a lot of signal out of that, but in the hands of a bad interviewer it would not work well.

Re: Harder programming questions do a worse job of predicting outcomes

#102

Post by stone-monkey »

gameswithgo wrote: post #56
warp_factor wrote: post #28

Earlier quoted context omitted.

This is why this whole interview thing is so absurd. The amount of days lost by engineers to relearn obscure algorithms and training on leetcode while we could be coding for things that are actually useful.

obscure algorithms are useful. it isn't a perfect system but its better than what most people propose as alternatives, which is to just have an ad hoc conversation. testing whether someone is willing to prepare for a thing is a relevant work skill test too.

>testing whether someone is willing to prepare for a thing is a relevant work skill test too.

Is it, though? I could understand if algo questions had some relation to the work you're doing, but your comment on why they're good is independent of the actual material. If we replaced the algorithm question interview with an interview testing obscure presidential facts, would it really be a useful test to have devs take? I guess it is a relevant work skill test in that you get people willing to put the work in/game the system, but it doesn't seem to be much of (if at all) an improvement from ad hoc conversations.

Re: Harder programming questions do a worse job of predicting outcomes

#103

Post by daenz »

zinclozenge wrote: post #15

The less talked about absurdity is that successfully passing programming interviews is a skill itself. It's especially absurd because the time I spend developing that skill is less time spent developing skills and knowledge more directly relevant to my job. Yet, programming interview skill is more relevant to progressing my career. edit: now if you'll excuse me, I need to do some dynamic programming problems.

This resonates strongly with me. I know a few engineers who are bad to mediocre software developers, but excellent interviewees. That skill alone lands them offers at any place they want to work. They can whiteboard algorithms like there is no tomorrow, but they can't manage software complexity.

Re: Harder programming questions do a worse job of predicting outcomes

#104

Post by mlthoughts2018 »

SilasX wrote: post #75
rightbyte wrote: post #24

Earlier quoted context omitted.

This is kind off strange for a non US resident to grasp. I've been employed as a programmer three times and noone tested my coding abilites what so ever on the interviews. No samples, nothing. Never heard of any collegue doing that either.

What stops someone unqualified from getting through? What other filters do you apply?

Probing someone with very technical conversation about past technical projects is a much stronger filter to prevent unqualified candidates than passing CoderPad tests, whiteboard algorithms, etc.

The conditional probability you are hopelessly lacking software skills to do a job given that you nonetheless passed a TripleByte exam or something is quite high. Overfitting & memorization for the sake of the test is extremely common.

But it’s much, much harder to fake competence when needing to dynamically and verbally explain technical details in a conversational interview about past work experience.

Re: Harder programming questions do a worse job of predicting outcomes

#105

Post by peteri »

dcplogic wrote: post #7

A while ago we had a job applicant who had travelled very far and long to reach us. (literally from the other side of the world.) So, as a courtesy we figured, why not spend a few hours extra with this applicant in the programming test. We set up a laptop with a clean Ubuntu install, devised a programming test that was quite involved. Not algorithmic hard, just more complex than what can normally be done within a 20-…

This can be problematic, you waste a lot of time setting up the PC / explaining the problem.

Last time I was involved with this interview style it always seemed to take an hour to get setup which meant a long interview of 3-4 hours particularly if a candidate went down the wrong path.

In the end we optimised for SOLID principles with a blackbox dll that had a function that slept for 2 seconds and a calling class that had mixed responsibilities (logging and calling the dll). We started folks off with a test or two and hoped they'd inject a mock to get rid of the delay and split logging off into a separate class.

I'm not saying it was a great test but you could do something within an hour or so then maybe spend half an hour talking through what techniques they'd use for a more complicated scenarios.

If you can afford the time then more realistic testing is great and I do think you should try.

Re: Harder programming questions do a worse job of predicting outcomes

#106

Post by avip »

joker3 wrote: post #101

There's a problem here. The only thing that Triplebyte can claim based on their data is that easier programming questions are more predictive of performance among candidates who received an offer . Since candidates who get offers are (in theory) different from candidates who don't get offers, we can't necessarily generalize from one population to the other. There's also a question about how to mix question difficulty…

Yep. Lot's of "data analysis" without even defining what would be a "false positive" (someone who did well in an interview, got an offer but turned out to be a bad hire? how is TB exposed to that critical data exactly?)

Re: Harder programming questions do a worse job of predicting outcomes

#107

Post by SomeHacker44 »

gameswithgo wrote: post #56
warp_factor wrote: post #28

Earlier quoted context omitted.

This is why this whole interview thing is so absurd. The amount of days lost by engineers to relearn obscure algorithms and training on leetcode while we could be coding for things that are actually useful.

obscure algorithms are useful. it isn't a perfect system but its better than what most people propose as alternatives, which is to just have an ad hoc conversation. testing whether someone is willing to prepare for a thing is a relevant work skill test too.

Knowing the existence of, and understanding the performance characteristics of a broad range of algorithms is, to my mind, a much more useful set of working knowledge than the details of any individual algorithm. The latter can always be looked up. It is harder to find out the former and much better to say “yes, there is an algorithm such-and-such that may be applicable here, let me spend 5 minutes checking that.” In my experience, anyway.

Re: Harder programming questions do a worse job of predicting outcomes

#108

Post by alchemism »

Zelphyr wrote: post #76

I see one aspect of this trend of asking programming questions that require a lot of memorization: We have had for the past ~10-15 years people in the workforce (and thus acting as interviewers) who went through a public education system where heavy emphasis was placed on passing tests that required a lot of memorization. I'd be interested to know how many of these interviewers actually think they're able to identify…

> Ultimately it seems like there is a soft element to interviewing that is being tossed out now, which is: do I think we can work with this guy/gal?

A colleague offered this simple heuristic for the soft side of [the] interview:

"If they are going to be equal or junior teammates, I ask myself, 'Would I feel comfortable sitting in a conference room with this person and hashing out a design or troubleshooting a problem for two hours?'

If I'm evaluating someone who is going to be senior to me, or my direct boss, I ask myself, 'Would I feel comfortable following this person's technical instructions if they handed them to me in a document?'"

In my personal professional opinion, this measure is more useful to an engineering org. than any "Stump the Chump" style technical-trivia screening.

Re: Harder programming questions do a worse job of predicting outcomes

#109

Post by TheOtherHobbes »

antibland wrote: post #26

A few jobs back I was tasked with hiring new developers to bolster a thin front-end team. The job was very CSS/JavaScript heavy, so I asked questions that were pertinent to what the candidate would be doing if hired. Of the five candidates, only one answered all the questions perfectly, and he turned out to be the biggest bust for us. The other candidates, after answering some of the harder questions incorrectly, see…

Personality has a huge amount to do with outcomes. Especially in teams, where the best teams have a mix of complementary talents.

If you're hiring for Generic Developer Skills you're going to get generic developers - and much less development than a more flexible approach would give you.

Re: Harder programming questions do a worse job of predicting outcomes

#110

Post by PopeDotNinja »

> However, whether or not a candidate answers a question correctly is not the only source of signal during an interview. You can also evaluate their process by, for example, observing how long it takes them to finish, how clean their code is, and how much they struggle while finding a solution. Our analysis shows that this second source of signal (process) is almost as predictive as the first (correctness).

I seem to do OK when an interviewer:

- asks me a question that sounds like a real problem, not a contrived one (although on occasion I'll have fun with a contrived puzzle if the interviewer has a sense of humor & makes the process light hearted)

- doesn't push me down a path that requires me to implement a "simpler" solution I'd never consider (e.g. asking me a question that clearly wants an O(N) solution & then pushes me to try the O(2^n solution first)

- talks like a person with a problem, and not as someone who clearly knows what they want & simply won't say it

- doesn't try to "see how I think", because I code as much in my head as I do on a screen, meaning most of the code I throw into a text editor is the latest thought in a stream of random ideas until I get to one that works

- doesn't constantly interrupt me

- states their actual expectations, such as "I don't expect you to finish, what I'm really looking for is X"

Post reply on HN