Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Harder programming questions do a worse job of predicting outcomes

triplebyte.com

Post reply on HN
Print view View original

241–250 of 540 posts

Re: Harder programming questions do a worse job of predicting outcomes

#241

Post by bunderbunder »

whatshisface wrote: post #179
on_and_off wrote: post #161

Earlier quoted context omitted.

It is a perverse effect on the profession as a whole. In the silicon valley I have a fair amount of colleagues that expect of any dev to spend a fair amount of their free time grinding on even more dev. No surprise that there is a lack of diversity in the profession as a result.

Except for people over 50 who have probably realized there is more to life, why would some demographics be willing to grind less than others? Edit: By "over 50," I mean age demographics in general, which is the main thing that raises your family obligations. The only demographic division that I can think of that would reduce someone's willingness to abandon their personal life would be age.

For a non-age-related one:

It's a lot more socially acceptable for members of one sex than it is for members of the other to be too busy with career stuff to spare much time for their families.

On a somewhat related note, single parents simply aren't going to have that kind of time.

Re: Harder programming questions do a worse job of predicting outcomes

#243

Post by KallDrexx »

tombert wrote: post #125
griffinkelly wrote: post #74

Earlier quoted context omitted.

I've worked with some folks who were decent engineers who originally didn't get offers from FANG companies because they blew the interview; they later studied their ass off on leetcode and got offers. This did not make them better engineers at all. Ultimately, its just studying for the test, very much like the ACT/SAT in high school. You can be great at taking tests but ultimately a terrible student or vice versa.

I go back and forth about how I feel about doing heavy algorithm/data-structure stuff in interviews. On the one hand, I have never once written any kind of sort algorithm or LRU cache by hand for a production system, because why would I? Pretty much every language's standard library has a fairly-optimized sort and caching thing built in, and if they don't then there's still probably a million outside libraries to do…

> On the other hand, I genuinely do feel theory is really important. While not knowing the minutia of a tim-sort doesn't indicate that you'll be a bad engineer, not knowing the runtime efficiency of a sort can lead to some really awful code. Not knowing when to use a hash table instead of a nested-for loop can be a sign that you don't really know what you're doing, and not knowing some rough theory on concurrency indicates that I might be stuck debugging your race conditions or deadlock.

Theory is important but what's more important is how someone applies the theory to actual problem solving.

Reversing a binary tree isn't testing your knowledge of theory as much as it's really just testing memorization of a very very specific application. Knowing how to reverse a binary tree or how a hash map works is pointless if the person can't identify when to use them when solving an actual higher level problem. No one is ever given a binary tree and told to reverse it in the real world, they are given a business problem that you identify can be solved efficiently by modelling it as a binary tree and reversing it.

I'd bet most of the good people who fail the "reverse a binary tree" type of questions would succeed if you give them a realistic problem to solve without forcing a very specific solution onto them. Either they will come up to the realization that the solution is to think of the problem in terms of binary tree or they won't.

And neither is a terribly bad answer either. If they recognize you gave them a problem that can be represented as a binary tree and efficiently solved by reversing it then there you go, not only did you prove they knew the "theory" but they knew how to apply it as well. If they don't recognize it you gain valuable insight into their line of thinking and they might find novel ways to represent the problem and apply other theoretical concepts that solve it efficiently (maybe more maybe less).

Re: Harder programming questions do a worse job of predicting outcomes

#244

Post by dahart »

My group likes starting with easy questions and ramping up the difficulty, not to eliminate people with wrong answers, but for two other reasons: first, to see whether people understand the questions and whether they try to make up answers, or ask questions, or say “I don’t know”. Second, to see what the limits & boundaries of their experience is. We know that people don’t know everything, and we measure more for potential than for knowledge, but it’s still useful to understand someone’s experience and exposure level.

More important than question difficulty to me is attitude, and I’d love to see whether attitude is measurable and how it compares to later performance, but curiosity and optimism and communication really do go further than right or wrong on math and engineer questions for me. That point might even be tired already, I know people say it all the time, but I’m going to keep saying it because we still have blog posts on question difficulty, when easy vs hard engineering questions are pretty low on my list of what matters when I’m hiring.

Re: Harder programming questions do a worse job of predicting outcomes

#245

Post by joshuamorton »

pcwalton wrote: post #79
gameswithgo wrote: post #56

Earlier quoted context omitted.

obscure algorithms are useful. it isn't a perfect system but its better than what most people propose as alternatives, which is to just have an ad hoc conversation. testing whether someone is willing to prepare for a thing is a relevant work skill test too.

Go search for "Knights on a Keypad" (a formerly-common Google interview question). Trying to imagine any situation in which the solution would be useful is harder than the problem itself.

You mean that you can't imagine when a memorized graph traversal would be useful?

I find that hard to believe.

Re: Harder programming questions do a worse job of predicting outcomes

#246

Post by gizmo686 »

ohyes wrote: post #208
bunderbunder wrote: post #120

Aside from all the things mentioned in the article, this also seems like a fairly predictable application of Goodhart's Law: "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes." Once upon a time, skill at doing these sorts of problems might have correlated (imperfectly) with general aptitude as a programmer or software engineer. But the very act of trying t…

What’s frustrating to me is that we haven’t found a way to teach programming that doesn’t rely on natural aptitude. It’s really a travesty that we can’t teach it the same way that we teach maths or natural languages. The end result is we’re left trying to divine whether someone is the programming equivalent of being illiterate. As with illiteracy people find ways to fake it.

As someone who has studied both math and computer science (and is now a professional programmer), I have no idea what you are talking about. We are far better at teaching people to program than we are at teaching them to do math. Its just that most people self select out of math [0], so, due to selection bias, it appears that we are great at teaching those that remain.

As an aside, as someone who also studied linguistics [1] (and a couple of foreign languages at a beginner level), I am very confident in saying that our approach to teaching natural languages relies almost entirely on natural aptitude. It is just that, absent a serious mental disorder, all humans have a very large natural aptitude for natural languages

[0] Probably because we are absolutly terrible at teaching it.

[1] I assume not what you mean by teaching natural languages, but the topic of language acuisition (including in adults) does come up; plus it gives some perspective on how teaching language would look if we didn't rely on natural aptitude.

Re: Harder programming questions do a worse job of predicting outcomes

#247

Post by [deleted] »

jorblumesea wrote: post #233
pmiller2 wrote: post #212

Earlier quoted context omitted.

The big difference is you pass the MCAT or bar exam once, then never worry about it again.

That's not true, not every state shares or accepts Bar results from other states, and Doctors have medical board exams that also need to be studied for. If you are a lawyer or a doctor and are moving to a new state, you probably need to study.

[deleted]

Re: Harder programming questions do a worse job of predicting outcomes

#248

Post by pmiller2 »

dudul wrote: post #163
pmiller2 wrote: post #112

Earlier quoted context omitted.

Homework is a bit of a catch-22. I agree that they can be made more real world than typical interview problems (although not all are). OTOH, why should I do your “4 hour” project, in which I’m competing with people who spend 8+ hours on it, just to get to the same onsite interview I could get with another company after a recruiter chat and a 45-60 minute technical phone screen? It’s not a good use of my time.

These are interesting points. Just to clarify first, the exercise should not take more than 1 hour. And I mean it. It is not difficult, or tricky or anything. It is used 1) to make sure the candidate can do very elementary things, 2) as support for follow up during the on-side interview. Regarding "getting the on site", if you don't do the exercise before you would get the "whiteboard" BS. And then you may be competi…

If your exercise truly takes 1 hour, and you're evaluating it on the basis of 1 hours' worth of effort, then fair enough. I would say that roughly equates to the technical phone screen in this case, except without all the annoying "let's talk about every little thing we're doing while we're doing it, so it takes twice as long" BS.

And there's nothing wrong with whiteboarding, if done appropriately. Whiteboarding a system design question is totally appropriate. Asking a candidate to write down code for finding the longest palindromic substring of an input string in linear time is not. I've never come to a situation where writing anything more than the barest pseudocode outline for what to do on a whiteboard was the best way to get a real world task accomplished.

Re: Harder programming questions do a worse job of predicting outcomes

#249

Post by msla »

whatshisface wrote: post #179
on_and_off wrote: post #161

Earlier quoted context omitted.

It is a perverse effect on the profession as a whole. In the silicon valley I have a fair amount of colleagues that expect of any dev to spend a fair amount of their free time grinding on even more dev. No surprise that there is a lack of diversity in the profession as a result.

Except for people over 50 who have probably realized there is more to life, why would some demographics be willing to grind less than others? Edit: By "over 50," I mean age demographics in general, which is the main thing that raises your family obligations. The only demographic division that I can think of that would reduce someone's willingness to abandon their personal life would be age.

> The only demographic division that I can think of that would reduce someone's willingness to abandon their personal life would be age.

I can think of a few others tied to national work culture.

Re: Harder programming questions do a worse job of predicting outcomes

#250

Post by alboy »

chooseaname wrote: post #177
gonehome wrote: post #94

Earlier quoted context omitted.

It feels very much like that to me which is why it’s strange there isn’t a Kaplan equivalent. There’s interview cake and leetcode, but I think people would pay $2k for a class that focuses on the questions and in person whiteboard practice. They could collect information about the interviews at the major companies and then use those to create the program. For payment could also help candidates negotiate and then take…

The only way I would consider paying for a course like this is if, upon successfully completing it, I would not have to do the technical portion of the interview at companies I applied to. They would accept this cert and just do the soft skills interview.

One could argue that CS undergraduate degrees are perfectly suited for this purpose. FAANG & Co don't seem to care for that. You've successfully passed a few dozen exams at MIT or Stanford? Well, let's make the reasonable assumption that you have no clue about algorithms and data structures and start with our third whiteboarding session, that should be a way better proxy for your knowledge.
Post reply on HN