Harder programming questions do a worse job of predicting outcomes • Hacker News Board

Re: Harder programming questions do a worse job of predicting outcomes

#111

Post by je42 » Mon, Feb 18, 2019, 7:35 PM UTC

I usually ask what's your strongest language; then ask questions about that programming language. f.e. if it is python:

> how would you explain the with statement to a junior developer ? then increasingly difficult questions that go into the language runtime/concepts.

one other favourite question of mine is:

> Imagine, you got a standard website the serves data from a database. When a customer types in the url into the browser bar what needs to happen until the customer see the website.

> Go as deep as you can in the answering the question.

When you got an answer, you'll see frontend engineers explain more about the browser, while backend engineers talk more about the backend.

There was one very senior engineer, that actually talked about the ethernet layer, he talked for more than 15min. Most medior engineers are done in 5mins. ;)

Re: Harder programming questions do a worse job of predicting outcomes

#112

Post by pmiller2 » Mon, Feb 18, 2019, 7:35 PM UTC

dudul wrote: post #42

This article makes a lot of good points. After going through the "implement a red-black tree on this whiteboard" experience as a more junior dev, I always promised myself I would never use this kind of stupid questions to hire. Now, 13 years later, I mostly rely on "homework" type exercises. I think they address most of the issues. They are more "real world", no time pressure, etc. However, even those now are being h…

Homework is a bit of a catch-22. I agree that they can be made more real world than typical interview problems (although not all are). OTOH, why should I do your “4 hour” project, in which I’m competing with people who spend 8+ hours on it, just to get to the same onsite interview I could get with another company after a recruiter chat and a 45-60 minute technical phone screen? It’s not a good use of my time.

Re: Harder programming questions do a worse job of predicting outcomes

#113

Post by avip » Mon, Feb 18, 2019, 7:35 PM UTC

davidhyde wrote: post #25

I think that asking a candidate to perform a code review can be an effective method of evaluating quite a few desirable qualities. Can they understand someone else's code? Can they engage in constructive critical discussion? Are they able to effectively refactor something to make it better? Can they spot mistakes and do they have an opinion about how to avoid such mistakes?

There's a classical test of giving a never seen before programming language (invented for the test), then the candidates need to reason about some code. Never seen that in the industry though.

Re: Harder programming questions do a worse job of predicting outcomes

#114

Post by Zelphyr » Mon, Feb 18, 2019, 7:36 PM UTC

joker3 wrote: post #101

There's a problem here. The only thing that Triplebyte can claim based on their data is that easier programming questions are more predictive of performance among candidates who received an offer . Since candidates who get offers are (in theory) different from candidates who don't get offers, we can't necessarily generalize from one population to the other. There's also a question about how to mix question difficulty…

You touch on a very important part of this equation: there are good interviewers and bad interviewers.

A good interviewer, in my opinion and experience, will try to get to know the person first. They'll put the candidate at ease. They want to try as much as possible to be talking to the person they're possibly going to be working with, not the anxious candidate that just walked into the room.

Re: Harder programming questions do a worse job of predicting outcomes

#115

Post by darpa_escapee » Mon, Feb 18, 2019, 7:36 PM UTC

paul7986 wrote: post #29

One interview of mine asked something i didn't know yet I said there's no doubt I figured it out via Google. The interview pretty much ended there and I'm glad it did! Any place or interviewer that says you shouldn't use Google of OverFlow to get your work done is no place I want to work for.

What, you couldn't derive a solution in a vacuum from first principles? I bet you think referring to API documentation isn't cheating, too!

Re: Harder programming questions do a worse job of predicting outcomes

#116

Post by mlthoughts2018 » Mon, Feb 18, 2019, 7:36 PM UTC

smt88 wrote: post #36

rightbyte wrote: post #24

Earlier quoted context omitted.

This is kind off strange for a non US resident to grasp. I've been employed as a programmer three times and noone tested my coding abilites what so ever on the interviews. No samples, nothing. Never heard of any collegue doing that either.

The massive FAANG-types of companies receive so many millions of job applications, they championed these types of interviews to have a more objective way to filter through all the applicants. That practice has waned somewhat at the larger companies (not totally, but it's changing), but has trickled down to smaller companies.

I received a cold call from a Lyft recruiter recently for a senior machine learning position. I asked why they were reaching out to me and the recruiter mentioned that it’s an especially hard time to locate experienced machine learning candidates, they don’t have enough applicants.

I said I was interested in interviewing but that I would only agree to a process that evaluates me based on my previous work history, and not any onsite or takehome coding projects, system design questions or whiteboard coding questions.

The recruiter said she would run it by the manager, but thought it would not be possible, and a few days later I got a rejection email.

Re: Harder programming questions do a worse job of predicting outcomes

#117

Post by nybblesio » Mon, Feb 18, 2019, 7:36 PM UTC

Zelphyr wrote: post #76

I see one aspect of this trend of asking programming questions that require a lot of memorization: We have had for the past ~10-15 years people in the workforce (and thus acting as interviewers) who went through a public education system where heavy emphasis was placed on passing tests that required a lot of memorization. I'd be interested to know how many of these interviewers actually think they're able to identify…

So much this. I've been programming for 30+ years. My brain only has so much cache space and it dumps frequently. Asking me questions that clearly are testing my ability to hold large amounts of data in my meat computer isn't testing my ability to design and write computer programs.

Sure, I'd love to have a "mind palace" like Sherlock. Alas, I do not. I often admit this as early in the interview process as possible to avoid wasting time.

Re: Harder programming questions do a worse job of predicting outcomes

#118

Post by alfalfasprout » Mon, Feb 18, 2019, 7:39 PM UTC

zinclozenge wrote: post #15

The less talked about absurdity is that successfully passing programming interviews is a skill itself. It's especially absurd because the time I spend developing that skill is less time spent developing skills and knowledge more directly relevant to my job. Yet, programming interview skill is more relevant to progressing my career. edit: now if you'll excuse me, I need to do some dynamic programming problems.

I've interviewed tons of engineers that ace the interview and you have no choice but to pass but you see later on writing abysmal code. The problem is that design, quality, and maintainability are not valued. As a result, you end up with extremely mediocre "hacky" engineers at larger companies that can spit out tons of valid code but lead to a tangled mess shortly thereafter.

Worse, there's this huge emphasis on "big O" with zero focus on clearly egregious bad practices (tons of copies, outrageous memory usage, casting between strings and numbers all the time). I've rarely seen clearly bad algorithms be deployed but I have seen plenty of unperformant code go out.

Re: Harder programming questions do a worse job of predicting outcomes

#119

Post by zinckiwi » Mon, Feb 18, 2019, 7:39 PM UTC

I like some coding in interviews. I try to make the tasks fairly real-world though. I am a UI engineer so my tasks typically fall into one of two buckets:

- A take home (2-3 hours) task for retrieving tabular data from an API and displaying it. Here I'm looking for general framework chops, readability, some design sense.

- An in-person (~45m) not-quite-pair programming task, with a real computer, tools, editor etc., for doing a typical UI operation, e.g. truncating text. Starts simple and gets more complex as time allows: make a function to truncate text to x chars; now add an ellipsis only if truncation occurred; now make sure not to truncate in the middle of a word, etc.

Re: Harder programming questions do a worse job of predicting outcomes

#120

Post by bunderbunder » Mon, Feb 18, 2019, 7:39 PM UTC

Aside from all the things mentioned in the article, this also seems like a fairly predictable application of Goodhart's Law: "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes."

Once upon a time, skill at doing these sorts of problems might have correlated (imperfectly) with general aptitude as a programmer or software engineer. But the very act of trying to leverage that correlation for hiring purposes probably also made it go away. Now you've got a whole lot of people practicing hard on these sorts of problems, spending huge chunks of their free time grinding away on Project Euler and Advent of Code and HackerRank. That muddies the quality of this stuff as a proxy for what it was originally trying to detect: natural aptitude. I'm guessing having time to level grind like that also correlates inversely with other traits that are desirable in a programmer.