Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We compiled a library of realistic engineering take-home tests and ranked them

trytapioca.com

Post reply on HN
Print view View original

131–140 of 190 posts

Re: We compiled a library of realistic engineering take-home tests and ranked them

#131

Post by alexyang21 »

fancy_pantser wrote: post #111
alexyang21 wrote: post #2

Studies [1] show that a work sample test is the best predictor of candidate performance on the job, which is why many software engineering teams use take-home tests as one step in their hiring process. But designing an effective test is difficult and time-consuming. For example, candidates are reluctant to complete tests that are too long or not engaging enough. But make them too short and teams won’t get the signal…

Why are there no 1-star projects listed?

The reason why there aren't any 1-star projects isn't because they don't exist, but because we didn't think anyone would want to see them. Cataloguing these was a ton of work (we sifted through hundreds of tests). Including the 1-star ones seemed like it would only shame companies who used them.

Re: We compiled a library of realistic engineering take-home tests and ranked them

#132

Post by twawaaay »

1123581321 wrote: post #36
twawaaay wrote: post #5

As an interviewer, I believe take home tasks are unfair and also poor return on investment. Take home tasks incentivise candidates to burn as much time as possible on the task. Consequently, they are a test of who is willing to put a week of work even when the instructions say it should not take more than 3 hours to complete. I prefer pair programming with the candidate on the premise that if I ask you to spend 2 hou…

A two hour pairing session is hard on people who need to fit their interviewing time into odd hours. It’s also stressful, because it’s so much more consequential than a session of real work with two peers. Not to say you shouldn’t do it—just keep in mind it also has trade-offs. And I soundly agree with the positives you list, especially that it forces the interviewer to commit time like the interviewee. I did a one h…

If you can't find 2h for a Zoom call I start questioning your commitment to get the job, especially when there is no time limit on when you can have the meeting (within reason). When I had an important interview in a place where I really wanted to work I would usually take entire day off and make sure I am well rested, prepared and in right state of mind.

Yes, it is stressful. I try to make interviews less stressful but I also believe ability to perform under stress is a desirable trait. I have been presented with many stressful situations in my past where I had to deal with high stakes, difficult problems on a short notice, with important people watching my every move. Like dealing with an outage that incapacitated an entire national bank and was causing a loss of about 50M USD per day.

Now, I am mostly interviewing senior devs, team leads, tech leads, etc. I would be much more lenient with junior roles but I just don't have time to interview everybody and I also like to give the chance for other people to work on their interviewing skills.

Re: We compiled a library of realistic engineering take-home tests and ranked them

#133

Post by rendall »

Once, when I started out, I did a take-home test and didn't get the job. The feedback was literally "In some of the files, you mixed tabs and spaces" - which was not true. The penny dropped a few years later: there never was a job. I did some kid's homework.

Re: We compiled a library of realistic engineering take-home tests and ranked them

#134

Post by NeverFade »

d23 wrote: post #110

These are great in theory. In practice, I've regretted the handful of times I've done them. It seems like some companies use this as a way to "waste" less of their interviewers' time. One particular dysfunctional company that everyone here would recognized gave me the feedback: you're one of the candidates out of nearly 200 that did this [7-10 hour take home test] and passed! Then they ghosted me for a month before a…

As a technical hiring manager, I was involved in some teams that did use homework assignments.

Their fatal issue on the hiring side is that they take too much time to properly score.

If you ask a good candidate to work for 4+ hours, they'll produce code that will cost 2+ hours of a great engineer to thoroughly check.

Lots of companies are very fond of giving candidates huge tasks and having them spend 4+ hours on these. But practically no company will have their best senior engineers (a very scarce resource) dedicate dozens of hours per month to check these tasks.

Hence, these tasks effectively never receive the resources required to properly check them. Especially at the more senior level. If you're hiring a senior engineer, and giving them a 4 hour homework task, a staff-level engineer would have to spend 2-3 hours to properly check it.

Forget about it, that's not happening.

Instead, these teams do ask the senior engineer to take 4 hours to work on the task. However, once the hard part arrives, the team will never check it properly. They'll assign some junior engineer for an hour to check it.

The team won't care, because it's the candidate's time being wasted, while theirs is efficiently preserved. Plus, our junior engineers are so amazing, they can certainly score the work of a senior in less than an hour.

Re: We compiled a library of realistic engineering take-home tests and ranked them

#135

Post by alexyang21 »

alfalfasprout wrote: post #65

I think a lot of people are getting hung up on the fact that these challenges take a long time. Yep, and that's a problem... But the goal of having a candidate work on a real problem is a good once. IMO a far better approach is to do a live exercise with them. Not a leetcode problem... take an existing codebase and add a feature to it, fix an issue with it, etc. Heck, you don't need them to complete everything, but y…

I'm a big fan of live exercises too, mainly because it's a great way to see how candidates think and how they collaborate with others. There are only a couple tradeoffs with this format (every approach has them): live exercises can be more stressful and teams with many applicants won't have the bandwidth to offer this to everyone who could be qualified.

My favorite is a combination: short (1 hr) take-home followed by live discussion/pairing with anyone who does a half-decent job. It reduces stress because candidates will already be familiar with the code (they wrote it!) while being efficient with time.

Re: We compiled a library of realistic engineering take-home tests and ranked them

#136

Post by yamtaddle »

ghaff wrote: post #66
mellavora wrote: post #29

Earlier quoted context omitted.

> Even 2-4 hours is more time than conventional Leetcode-style initial assessments, Hard disagree on this one. Unless your day-to-day work includes many leetcode style problems, you need to put in significant time training on leetcode if you want to pass the interview. You should be able to complete a take-home based on your current skill set. Yes, it may take 4-6 hours (or 8-10), and yes, that is a big ask of a cand…

Is there literally any other field where there's an expectation to essentially do fairly intense studying to pass interviews that apparently have very little to do with day to day jobs? Maybe they exist I've never encountered them. The limit of my "studying" for an interview is to just learn a bit about the company, their strategy, etc. either by online research or by talking to people I know connected with them. ADD…

> ADDED: I suppose one could argue that the Bar exam is a bit like that but that's a credential as are degrees which are not necessarily all that overlapping with the real world.

Yeah, I think most other professions where one might imagine them doing something like a stereotypical software interview, use that instead: a credential obtained by a (perhaps very difficult!) test, maybe with periodic re-tests or required refresher courses to keep the credential valid.

I have a suspicion that a big part of why top-comp software companies keep their interviews so incredibly unpleasant has more to do with discouraging job-hopping among them (so, suppressing wages) than with its being the best process for hiring good developers.

Re: We compiled a library of realistic engineering take-home tests and ranked them

#137

Post by alexyang21 »

anchochilis wrote: post #76
twawaaay wrote: post #5

As an interviewer, I believe take home tasks are unfair and also poor return on investment. Take home tasks incentivise candidates to burn as much time as possible on the task. Consequently, they are a test of who is willing to put a week of work even when the instructions say it should not take more than 3 hours to complete. I prefer pair programming with the candidate on the premise that if I ask you to spend 2 hou…

As a candidate, I vastly prefer a take-home to live whiteboarding or pairing. That background sensation of being observed and evaluated greatly interferes with my ability to concentrate and perform at my best. (I'll note that I have no problem pairing in real-world work scenarios! I love discussing problems with my team. It's the artificial pressure injected by the interview process that really gets to me)

How would a take-home compare to a hybrid format? For example, if you were given a 1-hour take-home followed by a live session asking about your thought process and then pairing to extend what you previously worked on.

Re: We compiled a library of realistic engineering take-home tests and ranked them

#138

Post by q7xvh97o2pDhNrh »

yamtaddle wrote: post #136
ghaff wrote: post #66

Earlier quoted context omitted.

Is there literally any other field where there's an expectation to essentially do fairly intense studying to pass interviews that apparently have very little to do with day to day jobs? Maybe they exist I've never encountered them. The limit of my "studying" for an interview is to just learn a bit about the company, their strategy, etc. either by online research or by talking to people I know connected with them. ADD…

> ADDED: I suppose one could argue that the Bar exam is a bit like that but that's a credential as are degrees which are not necessarily all that overlapping with the real world. Yeah, I think most other professions where one might imagine them doing something like a stereotypical software interview, use that instead: a credential obtained by a (perhaps very difficult!) test, maybe with periodic re-tests or required…

> I have a suspicion that a big part of why top-comp software companies keep their interviews so incredibly unpleasant has more to do with discouraging job-hopping among them (so, suppressing wages) than with its being the best process for hiring good developers.

You have to wonder if this has backfired. ;) Plenty of people job-hop every 1-2 years, and it's pretty common knowledge which companies ask which level of LC difficulty for which level.

Re: We compiled a library of realistic engineering take-home tests and ranked them

#139

Post by twawaaay »

anchochilis wrote: post #76
twawaaay wrote: post #5

As an interviewer, I believe take home tasks are unfair and also poor return on investment. Take home tasks incentivise candidates to burn as much time as possible on the task. Consequently, they are a test of who is willing to put a week of work even when the instructions say it should not take more than 3 hours to complete. I prefer pair programming with the candidate on the premise that if I ask you to spend 2 hou…

As a candidate, I vastly prefer a take-home to live whiteboarding or pairing. That background sensation of being observed and evaluated greatly interferes with my ability to concentrate and perform at my best. (I'll note that I have no problem pairing in real-world work scenarios! I love discussing problems with my team. It's the artificial pressure injected by the interview process that really gets to me)

It is not about what candidates prefer. If that was the case, there would be no whiteboards or any kind of exercises at all.

Re: We compiled a library of realistic engineering take-home tests and ranked them

#140

Post by q7xvh97o2pDhNrh »

alexyang21 wrote: post #131
fancy_pantser wrote: post #111

Earlier quoted context omitted.

Why are there no 1-star projects listed?

The reason why there aren't any 1-star projects isn't because they don't exist, but because we didn't think anyone would want to see them. Cataloguing these was a ton of work (we sifted through hundreds of tests). Including the 1-star ones seemed like it would only shame companies who used them.

I appreciate that you might not want to name-and-shame. I wonder if you could just scrub the identifying info and post them, though?

Personally I think it'd still be tremendously valuable to show clear examples of what "bad" looks like in your ranking system. Not only does it help you show off more of the hard work you did, but it also helps all your readers get a more complete understanding of your framework.

Beyond that, it's generally helpful -- in learning anything complex -- to see examples of "what not to do, and why."

Post reply on HN