Harder programming questions do a worse job of predicting outcomes • Hacker News Board

Re: Harder programming questions do a worse job of predicting outcomes

#261

Post by [deleted] » Mon, Feb 18, 2019, 9:03 PM UTC

thoman23 wrote: post #240

davidkellis wrote: post #166

Earlier quoted context omitted.

I agree with the sentiment, but I noticed recently that TaxJar (I considered applying there at one point) mandates a trial period along these lines. From https://life.taxjar.com/distributed-team-hiring-process/ , "... We hope a candidate is able to spend somewhere between 80 and 160 hours working with TaxJar during their trial. ..." Presumably there are some people who are willing to do a trial, either concurrent wit…

Thinking more about TaxJar...I wonder if they are at any risk of employees of an existing firm doing work for TaxJar on the current company's time and equipment, which would in many cases make the TaxJar code the intellectual property of the former company.

[deleted]

Re: Harder programming questions do a worse job of predicting outcomes

#262

Post by walamaking » Mon, Feb 18, 2019, 9:04 PM UTC

davidkellis wrote: post #166

selestify wrote: post #137

Earlier quoted context omitted.

Who has time to spend a few weeks off from their full time job just to interview?

I agree with the sentiment, but I noticed recently that TaxJar (I considered applying there at one point) mandates a trial period along these lines. From https://life.taxjar.com/distributed-team-hiring-process/ , "... We hope a candidate is able to spend somewhere between 80 and 160 hours working with TaxJar during their trial. ..." Presumably there are some people who are willing to do a trial, either concurrent wit…

This company sounds terrible. You have to be hazed by all the other brogrammers at the end after doing 80-160 hrs of unpaid work, ridiculous:

https://life.taxjar.com/taxjar-traditions/

Re: Harder programming questions do a worse job of predicting outcomes

#263

Post by Gimpei » Mon, Feb 18, 2019, 9:05 PM UTC

esoterica wrote: post #257

Why do developers complain so much about hard interview questions that can be supposedly be gamed by studying to the test? Every high paying industry heavily engages in gatekeeping, because the number of people who want to make 400k/year is far larger than the number of 400k/year jobs available. The traditional forms of gatekeeping involve requiring that people have the right personal/familial connections, or have an…

I think because some people can't train themselves up through the gate. This is not because they can't figure out the leetcode problems, but because they have more difficulty dealing with the kind of pressure you see in interviewing situations than others.

Re: Harder programming questions do a worse job of predicting outcomes

#264

Post by esoterica » Mon, Feb 18, 2019, 9:05 PM UTC

CoolGuySteve wrote: post #259

esoterica wrote: post #215

Earlier quoted context omitted.

That's a nonsensical line of reasoning. Hard interviews don't reduce salaries, they increase them. Why do you think FAANG companies pay developers 300k+/year when there are so many lower-status companies who pay <100k/year and still manage to find hires? Because the latter are willing to hire candidates who have worse resumes and less impressive interview skills that the FAANG companies consider beneath them.

I suggest you read up on the effects of market friction. 300k/year could very well be a bargain for these people even if it sounds like a lot to you.

You might want to re-read your own post because whether 300k/year is a lot of money or not is a non sequitor. You claimed that companies ask hard interview questions to depress salaries, which is a trivially falsifiable claim when you note that companies that don't ask hard interview questions can get away with paying less money than companies that do because they are less picky about who they hire.

Re: Harder programming questions do a worse job of predicting outcomes

#265

Post by wolfgke » Mon, Feb 18, 2019, 9:05 PM UTC

"Easier interview questions are also less stressful, which is an important advantage. Stress causes candidates to under-perform. But, on the other hand, when candidates are more comfortable, they perform their true best, which actually makes interviews more predictive."

I cannot believe that any interview situation is comfortable.

Re: Harder programming questions do a worse job of predicting outcomes

#266

Post by glangdale » Mon, Feb 18, 2019, 9:05 PM UTC

on_and_off wrote: post #161

bunderbunder wrote: post #120

Aside from all the things mentioned in the article, this also seems like a fairly predictable application of Goodhart's Law: "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes." Once upon a time, skill at doing these sorts of problems might have correlated (imperfectly) with general aptitude as a programmer or software engineer. But the very act of trying t…

It is a perverse effect on the profession as a whole. In the silicon valley I have a fair amount of colleagues that expect of any dev to spend a fair amount of their free time grinding on even more dev. No surprise that there is a lack of diversity in the profession as a result.

I'm pretty sure it's age discrimination plain and simple. The only time in my life I would have been in great shape for the standard interview process without a good deal of study was a couple years in early graduate school, where nearly all of this "breadth" stuff was fresh in my mind.

The relentless scepticism about people's achievements is to some extent understandable (we've all run into the senior person who can't do fizzbuzz), but it ties neatly in with the idea that every new hire should be 25 at most.

Re: Harder programming questions do a worse job of predicting outcomes

#267

Post by ummonk » Mon, Feb 18, 2019, 9:05 PM UTC

minimaxir wrote: post #90

ummonk wrote: post #54

Earlier quoted context omitted.

How is that nonrealworld? Dealing with things like endianness is quite normal for a low level programmer.

Nonrealworld for the positions TripleByte typically hires for, anyways. It proxies the reader having a CS degree, which is incidentally what TripleByte's testing is supposed to deemphasize.

I don't think there would be too much correlation between CS degree and ability to answer that question. There are lots of CS degree programs that don't deliver such low level instruction, and there are plenty of people without CS degrees who can answer that question because they were curious about how computers work or have been exposed to low level programming.

Re: Harder programming questions do a worse job of predicting outcomes

#268

Post by entee » Mon, Feb 18, 2019, 9:07 PM UTC

Apocryphon wrote: post #190

tombert wrote: post #125

Earlier quoted context omitted.

I go back and forth about how I feel about doing heavy algorithm/data-structure stuff in interviews. On the one hand, I have never once written any kind of sort algorithm or LRU cache by hand for a production system, because why would I? Pretty much every language's standard library has a fairly-optimized sort and caching thing built in, and if they don't then there's still probably a million outside libraries to do…

> Not knowing when to use a hash table instead of a nested-for loop On the flip side, the O(1) look-up nature of hash tables make them the no-brainer data structure to use, at least for passing programming interviews. Perhaps more interesting test questions would be when not to use hash tables.

Fully agreed. Most questions I got when I started out as a data scientist interviewing for jobs were pretty simple: "use a hash table to count stuff, join stuff, lookup stuff". However not once did I get asked, "why do databases not exclusively use hash tables if they're so good?" That's a much more interesting question, though perhaps out of scope for a data scientist. I'd add that I've never heard that question being asked of engineers, I'd love to hear of people out there getting it though.

Re: Harder programming questions do a worse job of predicting outcomes

#269

Post by zanny » Mon, Feb 18, 2019, 9:09 PM UTC

ohyes wrote: post #208

bunderbunder wrote: post #120

Aside from all the things mentioned in the article, this also seems like a fairly predictable application of Goodhart's Law: "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes." Once upon a time, skill at doing these sorts of problems might have correlated (imperfectly) with general aptitude as a programmer or software engineer. But the very act of trying t…

What’s frustrating to me is that we haven’t found a way to teach programming that doesn’t rely on natural aptitude. It’s really a travesty that we can’t teach it the same way that we teach maths or natural languages. The end result is we’re left trying to divine whether someone is the programming equivalent of being illiterate. As with illiteracy people find ways to fake it.

I don't believe for a second it requires "natural aptitude" to program. The problem is any programming curriculum starts with a text editor open.

If a developer-to-be doesn't understand the framing context of what they are doing they are being dropped in a lake with no sense of direction.

Its why all the "naturals" started as geeks who played with computers from a young age. You learned about the environment you would end up working in and later on when you hit the grindstone and actually started creating gears to stick on that machine you had an idea what the result should look like and knew the tools in the shop when you set out to start building it. Even if you didn't know the steps involved in the process, you were familiar with the environment.

People who haven't spent time engrossed in computers, such as the myriads of youth entering a cs 101 class thinking its an easy career when the most exposure to tech they have had is maybe updating their phone and using apps for Facebook and Twitter and maybe owned a video game console with no tinkerability as a total black box drop out so fast. Their professors lead them to an anvil and tell them to forge a steel rod without any wink of an idea what a hammer is.

Its just not an answer anyone wants to hear, because the solution is only to have what amounts to an entire degrees worth of learning to predicate the actual study of programming. But you don't want your brain surgeon to go to medical school after having never studied high school biology or even more generally learned to read.

Re: Harder programming questions do a worse job of predicting outcomes

#270

Post by joshuamorton » Mon, Feb 18, 2019, 9:09 PM UTC

mlthoughts2018 wrote: post #104

SilasX wrote: post #75

Earlier quoted context omitted.

What stops someone unqualified from getting through? What other filters do you apply?

Probing someone with very technical conversation about past technical projects is a much stronger filter to prevent unqualified candidates than passing CoderPad tests, whiteboard algorithms, etc. The conditional probability you are hopelessly lacking software skills to do a job given that you nonetheless passed a TripleByte exam or something is quite high. Overfitting & memorization for the sake of the test is extrem…

I can, with quite a high level of competence, speak about projects I had no part in implementing or that don't yet exist.

It's far, far easier to fake expertise when you have more context than the person asking questions. Technical interview questions make sure that the question giver has more context than the recipient, ignoring pathological cases.