Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Harder programming questions do a worse job of predicting outcomes

triplebyte.com

Post reply on HN
Print view View original

231–240 of 540 posts

Re: Harder programming questions do a worse job of predicting outcomes

#231

Post by rajeshmr »

pmiller2 wrote: post #209
rajeshmr wrote: post #180

Earlier quoted context omitted.

This is good to know, typically it should be in concurrent with the existing employment. If we plan to switch companies, we commit time apart from existing office work for interview prep anyway. This interview prep time is what we would end up committing to work on their tasks etc. I think working on a project or task from the prospective employer would be much fun than preparing / memorizing interview questions etc.…

Most companies won’t allow such a thing, especially if the other company is a competitor. In any case, doing so would likely require me to take 1-4 weeks off from my regular job. Again, why would I want to blow all my vacation time working, just for the possibility of another job?

Okay my experience was, i didn't have to take leaves from my regular work. I had to simply dedicate a couple of hours post office time, to work on the task (the couple of weeks was given owing to time constraints due to my regular work). Post office hours, i would have to anyway commit to interview prep if i had decided to switch anyway is what i meant to say.

In any case, this is not the only method or approach that is the better alternative :) Maybe it will work for some and not for others. But evaluating alternatives is a good exercise. Currently we are stuck with this method because we do not have a better solution to the hiring problem.

Re: Harder programming questions do a worse job of predicting outcomes

#232

Post by gonehome »

austenallred wrote: post #157
gonehome wrote: post #94

Earlier quoted context omitted.

It feels very much like that to me which is why it’s strange there isn’t a Kaplan equivalent. There’s interview cake and leetcode, but I think people would pay $2k for a class that focuses on the questions and in person whiteboard practice. They could collect information about the interviews at the major companies and then use those to create the program. For payment could also help candidates negotiate and then take…

> If this isn’t part of lambda school already I think it should be It’s part of classes for our existing students, but it isn’t a single offering. One day.

Cool - if you need somebody to help I’d be interested.

Re: Harder programming questions do a worse job of predicting outcomes

#233

Post by jorblumesea »

pmiller2 wrote: post #212
jorblumesea wrote: post #182

Earlier quoted context omitted.

I mean, this is literally how any kind of industry with a test develops. Do you think the MCATs or the Bar exam are flawed because they have books on how to do well? Whenever there is any kind of standardization for testing, some ecosystem will evolve around it.

The big difference is you pass the MCAT or bar exam once, then never worry about it again.

That's not true, not every state shares or accepts Bar results from other states, and Doctors have medical board exams that also need to be studied for.

If you are a lawyer or a doctor and are moving to a new state, you probably need to study.

Re: Harder programming questions do a worse job of predicting outcomes

#234

Post by trhway »

ohyes wrote: post #208
bunderbunder wrote: post #120

Aside from all the things mentioned in the article, this also seems like a fairly predictable application of Goodhart's Law: "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes." Once upon a time, skill at doing these sorts of problems might have correlated (imperfectly) with general aptitude as a programmer or software engineer. But the very act of trying t…

What’s frustrating to me is that we haven’t found a way to teach programming that doesn’t rely on natural aptitude. It’s really a travesty that we can’t teach it the same way that we teach maths or natural languages. The end result is we’re left trying to divine whether someone is the programming equivalent of being illiterate. As with illiteracy people find ways to fake it.

using literacy model - we're testing people to write haiku according to all the rules of style and form when the job we're testing for is basically just low level clerical like copying documents (using copy machine), sometimes taking notes and filling simple forms.

Re: Harder programming questions do a worse job of predicting outcomes

#235

Post by Zimahl »

wiremine wrote: post #132

At my consultancy we recently streamlined our interview process: 1. Phone screen which takes 15 or 20 minutes. 2. The candidate fills out an essay, including showing us some code they're proud of. 3. If the essay ticks the boxes we conduct a 1 hour on site interview. We use the same a set of questions for every candidate, so the investment is easy to manage, and our team has a shared set of expectations on what is go…

If you want to do it for junior developers, fine, but as a senior dev, I'm not doing any take home assignment. It's an immediate pass from me. There is very little possibility that you can glean anything from my code other than picking apart it for code review bullshit. I'm not going to waste 2 to 6 hours on this - done it before, never again.

Re: Harder programming questions do a worse job of predicting outcomes

#236

Post by walamaking »

avip wrote: post #106
joker3 wrote: post #101

There's a problem here. The only thing that Triplebyte can claim based on their data is that easier programming questions are more predictive of performance among candidates who received an offer . Since candidates who get offers are (in theory) different from candidates who don't get offers, we can't necessarily generalize from one population to the other. There's also a question about how to mix question difficulty…

Yep. Lot's of "data analysis" without even defining what would be a "false positive" (someone who did well in an interview, got an offer but turned out to be a bad hire? how is TB exposed to that critical data exactly?)

It wasn't clear at all in the article what it even meant when it referred to "final outcome".

Re: Harder programming questions do a worse job of predicting outcomes

#237

Post by jgamman »

whatshisface wrote: post #213
sunir wrote: post #191

Earlier quoted context omitted.

They have kids. Developing a sentient being is a higher priority and more difficult than developing React Redux.

Wouldn't that just be another expression of age demographics?

some people have kids in their 20s (gasp!)

Re: Harder programming questions do a worse job of predicting outcomes

#238

Post by joshuamorton »

gedy wrote: post #220
tombert wrote: post #125

Earlier quoted context omitted.

I go back and forth about how I feel about doing heavy algorithm/data-structure stuff in interviews. On the one hand, I have never once written any kind of sort algorithm or LRU cache by hand for a production system, because why would I? Pretty much every language's standard library has a fairly-optimized sort and caching thing built in, and if they don't then there's still probably a million outside libraries to do…

> On the other hand, I genuinely do feel theory is really important. My beef with this in interviewing is that a huge chunk of modern programming work is developing UIs - and user interface theory and skills are treated like some softball thing. You ask UI-related questions to interviewers and half the time you get some shrug "oh we use whatever", etc.

This depends highly on the domain. I'm the only person on my team with any level of frontend work in the past ~2 years. And when I did, I was able to consult with UX experts and designers.

I can't say that theory is more universal than frontend, though anecdotally, for me, it is. But ui isn't universal.

Re: Harder programming questions do a worse job of predicting outcomes

#239

Post by pmiller2 »

rajeshmr wrote: post #187
pmiller2 wrote: post #178

Earlier quoted context omitted.

I still wouldn’t do it. Why should I, when I can do the entire interview process with another company with only a day’s worth of time expenditure on my end? If the task itself takes more than an hour or so, I can get to an onsite interview with far less effort than that by doing a recruiter chat and a phone screen.

Sounds fair enough :) Agreed. I have just proposed one of the alternatives which i could imagine / think of from the top of my mind. The problem isn't solved yet - or rather solved well yet. We could always brainstorm on a better alternative. :)

Here's the problem: you're dealing with a multiple prisoners' dilemma situation here. Theoretically, the best solution for all (candidates and companies) is to have a thorough, standardized, and real-world based interview process. If every company cooperated, then you could have that. But, when there are companies out there that will bring candidates onsite for ~5 hours after a little more than an hour's time commitment, that means I could theoretically do 5 of those interviews in a week. No other company is going to start demanding multiple days of an interviewee's time given that every other company's process takes ~1 day of active participation by the candidate. So, you need to limit your "thorough, standardized, and real-world" process to taking no more than a day.

Here's a process that might work, that I actually would participate in:

1. ~20 minute recruiter chat to establish basic levels of fit/compatibility.

2. ~45-60 minute phone screen.

3. Onsite consisting of 1 hour behavioral/cultural interview, and 3-4 hour programming task, preferably based on a real problem or task that arose in the company's code base, suitably extracted, simplified, and scrubbed of proprietary information. Internet resources are allowed, as is asking any questions of the interviewer, without penalty.

You could bring me in at 11 AM, do the behavioral interview, take me to lunch, then have the afternoon to do the programming task, and have me out by 4.

Re: Harder programming questions do a worse job of predicting outcomes

#240

Post by thoman23 »

davidkellis wrote: post #166
selestify wrote: post #137

Earlier quoted context omitted.

Who has time to spend a few weeks off from their full time job just to interview?

I agree with the sentiment, but I noticed recently that TaxJar (I considered applying there at one point) mandates a trial period along these lines. From https://life.taxjar.com/distributed-team-hiring-process/ , "... We hope a candidate is able to spend somewhere between 80 and 160 hours working with TaxJar during their trial. ..." Presumably there are some people who are willing to do a trial, either concurrent wit…

Thinking more about TaxJar...I wonder if they are at any risk of employees of an existing firm doing work for TaxJar on the current company's time and equipment, which would in many cases make the TaxJar code the intellectual property of the former company.
Post reply on HN