Harder programming questions do a worse job of predicting outcomes
1–10 of 540 posts
Re: Harder programming questions do a worse job of predicting outcomes
#2Another clarification: this data only applies to general programming exercises and not domain-specific knowledge. For example, if your role requires bio-informatics knowledge (or ML or AI etc.), then by all means ask about bio-informatics—even if it means asking harder questions. (We do, however, recommend keeping the content of general programming exercises as vanilla as possible, so that you don't filter out someone because they lack mastery of a subject that you don't actually intend to measure.)
Re: Harder programming questions do a worse job of predicting outcomes
#3,,Hard questions do filter out bad engineers, but they also filter out good engineers (that is, they have a high false-negative rate). Easy questions, in contrast, produce fewer false-negatives but more false-positives''
The philosophy at Google is that it's better to filter out 3 good engineers than to let in a bad one. The consequence of this is that it's really hard to get kicked out of Google.
The other part (whether it's more important to work on long easier questions to see how the candidate works on a large code base) is orthogonal reasoning, and that part may be true, depending on what type of engineers somebody is looking for.
Re: Harder programming questions do a worse job of predicting outcomes
#4Because they're designed that way. It all comes down to:
* Rejecting far more candidates than you need to -- so you can feel like you're hiring "the top 1 percent"
* Giving yourself the feeling that you have an objective hiring process (when really you don't)
* Making your own team members feel like they're super brilliant and special when really they're not
That's what the modern hiring process is designed to do. And in fact it works quite well, to serve this purpose.
Re: Harder programming questions do a worse job of predicting outcomes
#5Specific examples of what classifies as a “hard” or “easy” interview question would be very helpful to have reference points and assess one’s own interview process.
Re: Harder programming questions do a worse job of predicting outcomes
#6Instead of doing this, read, then follow https://sockpuppet.org/blog/2015/03/06/the-hiring-post/
Re: Harder programming questions do a worse job of predicting outcomes
#7A while ago we had a job applicant who had travelled very far and long to reach us. (literally from the other side of the world.)
So, as a courtesy we figured, why not spend a few hours extra with this applicant in the programming test. We set up a laptop with a clean Ubuntu install, devised a programming test that was quite involved. Not algorithmic hard, just more complex than what can normally be done within a 20-minute whiteboard interview. We expected it to take at least 2-3 hours. Google/Stack overflow/etc access was allowed and encouraged. "Just act as like you would normally do when solving a problem."
We spent like 2x4 hours devising this problem, based on our codebase (cutting out something somewhat easily digestible and making it able to run standalone).
It took like one hour to get productive. Explaining the problem, setting up editors, compilers, etc.
We took turns, but most of the time someone in the interview team (of two) sat next to the guy. We did give him some alone time.
This is probably nothing new in terms of interviewing techniques, but to us it was such a revelation. We learned so much more about the applicant. Perhaps it worked well with this guy because he happened to be a bit more outgoing than our typical successful applicant. We'd never felt so confident about giving someone an offer before.
I'm really looking forward towards testing out this approach with local candidates to see if we can replicate this "data gathering success".
Re: Harder programming questions do a worse job of predicting outcomes
#8I've never been giving given a practical programming test. Nearly all are useless algorithms. Create a function that takes a integer greater than 0. Build an array equal in size to the argument. Fill the array with numbers that when summed together equal zero.
It was worded far worse than that. What exactly is that telling you about the engineer?
On the flip side I'm asked to code full fledged applications but not to spend too much time on them... okay...
Another time I was asked to code a luhn algorithm. Oh and do it while a room of people watches you on giant screen cause that's what your day to day job will look like... I failed miserably and still got the job. What?!?!?
Re: Harder programming questions do a worse job of predicting outcomes
#9Specific examples of what classifies as a “hard” or “easy” interview question would be very helpful to have reference points and assess one’s own interview process.
Easy - reverse a string, determine if a string is a palindrome, reverse the digits of an integer, determine if one string is an anagram of another.
Hard - implement a subset of regex match in optimal time+space, find the operations required to turn 1 word into another word given a list of transitory words, find the median of 2 sorted arrays in optimal time, find the next permuted value.
Re: Harder programming questions do a worse job of predicting outcomes
#10,,Hard questions do filter out bad engineers, but they also filter out good engineers (that is, they have a high false-negative rate). Easy questions, in contrast, produce fewer false-negatives but more false-positives'' The philosophy at Google is that it's better to filter out 3 good engineers than to let in a bad one. The consequence of this is that it's really hard to get kicked out of Google. The other part (whe…
It's at least good that they call it a philosophy.