Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Has there been any academic research into software interviewing?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 129 posts

Re: Ask HN: Has there been any academic research into software interviewing?

#31

Post by d0mine »

onion2k wrote: post #12
dominotw wrote: post #7

Earlier quoted context omitted.

I really think this is best we got so far.

So the secret to being a 10X programmer is ":set tw=8" making your maximum line length 8 characters, forcing you be 10 times more productive than rival devs who're stuck at the old 80 character standard. Got it.

Or you can move around code creating large diffs and when asked why: just say "I'm familiarizing myself with the code" -- real story: my boss says about that person that he is the most productive programmer.

It seems gaming trivial metrics works in this case.

Re: Ask HN: Has there been any academic research into software interviewing?

#32

Post by barry-cotter »

I doubt there’s been too much on software interviewing specifically but interviewing/candidate selection for jobs is a large part of personnel psychology. A recent meta-analysis of what they know is below. Basically general mental ability (g, IQ) is the single best predictor, work sample tests work well, structured interviews can increase the performance of IQ tests or work samples and unstructured interviews are a trash fire.

> The Validity and Utility of Selection Methods in Personnel Psychology: Practical and Theoretical Implications of 100 Years

> This article summarizes the practical and theoretical implications of 85 years of research in personnel selection. On the basis of meta-analytic findings, this article presents the validity of 19 selection procedures for predicting job performance and training performance and the validity of paired combinations of general mental ability (GMA) and the 18 other selection procedures. Overall, the 3 combinations with the highest multivariate validity and utility for job performance were GMA plus a work sample test (mean validity of .63), GMA plus an integrity test (mean validity of .65), and GMA plus a structured interview (mean validity of .63). A further advantage of the latter 2 combinations is that they can be used for both entry level selection and selection of experienced employees. The practical utility implications of these summary findings are substantial. The implications of these research findings for the development of theories of job performance are discussed.

https://www.semanticscholar.org/paper/The-Validity-and-Utili...

Re: Ask HN: Has there been any academic research into software interviewing?

#33

Post by duxup »

sampo wrote: post #2

To correlate with job success, you need to define and measure job success. A sub-question: How's the academic research into measuring programmer job success?

Related, how do you define even the type of job and compare the person hired to the job?

A noob could be successful hire and not ultra productive compared to say a more experienced hire.

And that doesn't account for all the places hiring / think they need / hired "rockstar" developers .... to maintain their crud app.

Re: Ask HN: Has there been any academic research into software interviewing?

#34

Post by jrumbut »

sampo wrote: post #2

To correlate with job success, you need to define and measure job success. A sub-question: How's the academic research into measuring programmer job success?

I've always thought that surveys of team members and managers could be a pretty cheap and effective way to do this (in a research setting, where you aren't basing pay or contract renewal on it). It may not give you much in an absolute sense, but should give you a good idea of the relative ranking of programmers.

Of course software engineering research in general has always been a little unimpressive because what you really need is several teams completing projects, some using method A and others using method B. This could cost millions, and would only settle the bet between methods A and B.

Lacking this, we're left with the publically available research that relies on unconvincing proxies for success and very small samples. It's better than nothing but far short of what we ought to have given how important software is.

Re: Ask HN: Has there been any academic research into software interviewing?

#35

Post by artemisyna »

I know of at least one big tech company that has and it informs the ratio of different types of interviews for each level.

I imagine it'd be really hard for an academic research institute to do so. Firstly, they'd have to focus on technical interviewing in specific. Secondly, they'd have to get some big tech companies to share information about trajectory and whatnot for employees once they've been hired -- I imagine the extra work of building those pipelines/anonymizing the data/etc probably wouldn't be worth it for a lot of companies.

Re: Ask HN: Has there been any academic research into software interviewing?

#36

Post by JamesBarney »

sampo wrote: post #2

To correlate with job success, you need to define and measure job success. A sub-question: How's the academic research into measuring programmer job success?

Usually they look at future reviews or just ask managers how well that individual is performing.

Re: Ask HN: Has there been any academic research into software interviewing?

#37

Post by paulkon »

emaginniss wrote: post #26
lemony_fresh wrote: post #9

When I was self-studying social psychology I came across social facilitation theory and found it applicable to understanding interviewing. People have a hard time performing complex tasks while other people are watching because their working memory is monitoring the social situation. This varies for each individual. For tasks that have been rehearsed, like a musical or dance routine, they perform better. Politeness t…

I disagree completely about your theory. I have been on both ends of the whiteboard interview an almost uncountable time during my career. What it has always come down to in my mind is a collaborative process to solve a problem where the one performing the task is trying to solve the problem, while the giver of the task makes suggestions and points out weaknesses before they get out of control. I'm sure there are sad…

In that case, why use a whiteboard instead of a collaborative text editor like coderpad, repl.it, or even gnu screen?

Testing in an environment completely different from 'production' isn't going to give a super strong signal on collaborative problem solving imo.

Re: Ask HN: Has there been any academic research into software interviewing?

#38

Post by Scoundreller »

ghaff wrote: post #19
Scoundreller wrote: post #10

Not software development, but professional schools (eg medical schools) have published research about which admission selection methods can have predictive value both in-school and as a clinician. Lots of long follow-up. I’ve seen some programs do away with references, resumes and even interviews. The programs that I’ve seen remove interviews usually did replace them with standardized stations with different scenario…

I haven't looked for any recent research in this area but when I saw some way back when it was, in one sense, somewhat depressing. I forget if it was for undergrad or MBA programs but, as I recall, the correlations were basically to SATs/GMATs and class rank/grades. Of course, your results will vary depending on how you define success but basically quantitative hard measures had the most predictive value.

But what outcome were they measuring? Grades and rank in the MBA program?

A good student will probably continue to be a good student.

But how to select which students will become good practitioners may be a different story.

Re: Ask HN: Has there been any academic research into software interviewing?

#39

Post by sarbaz »

emaginniss wrote: post #26
lemony_fresh wrote: post #9

When I was self-studying social psychology I came across social facilitation theory and found it applicable to understanding interviewing. People have a hard time performing complex tasks while other people are watching because their working memory is monitoring the social situation. This varies for each individual. For tasks that have been rehearsed, like a musical or dance routine, they perform better. Politeness t…

I disagree completely about your theory. I have been on both ends of the whiteboard interview an almost uncountable time during my career. What it has always come down to in my mind is a collaborative process to solve a problem where the one performing the task is trying to solve the problem, while the giver of the task makes suggestions and points out weaknesses before they get out of control. I'm sure there are sad…

> it has never been what I've put people through.

How can you tell? Is this always based on how you experienced the interview, or do you ask candidates afterwards how they felt?

(no insult intended, but) you made an empirical statement, and I'm curious where your evidence is from.

Re: Ask HN: Has there been any academic research into software interviewing?

#40

Post by [deleted] »

JamesBarney wrote: post #36
sampo wrote: post #2

To correlate with job success, you need to define and measure job success. A sub-question: How's the academic research into measuring programmer job success?

Usually they look at future reviews or just ask managers how well that individual is performing.

[deleted]
Post reply on HN