We compiled a library of realistic engineering take-home tests and ranked them • Hacker News Board

Re: We compiled a library of realistic engineering take-home tests and ranked them

#151

Post by thisiswrongggg » Thu, Aug 25, 2022, 9:19 PM UTC

Unless I'm desperate I'll never do an unpaid take home test again in my life. For me it has been a waste of time and hugely demoralizing. Best/worst story and the one that broke camel's back: at one uber-copycat company I had my test failing on one reviewer. Then after a few months another reviewer interviewing me for another company told me that "this test would be fine by me. Had I reviewed it when I was back at the uber-copycat company I'd pass it".

And my experience was more or less like that with all unpaid take home tests. So, bitter truth is I have more to my life than doing boring unpaid work with minimal chances of getting to an offer and TBH I doubt I'd enjoy working in a company that hires like that. So, my stance is give me as much whiteboard or pair programming or whatever _time_bounded_ test you want and let's go from there.

PS: I looked one of the tests. It said "follow SOLID". I'm not gonna say what I think of SOLID here but it's good that they say what they adhere to beforehand. I mean if you're going to have people pouring hours and days in your tests at least be transparent of what exactly you expect.

Re: We compiled a library of realistic engineering take-home tests and ranked them

#152

Post by alexyang21 » Thu, Aug 25, 2022, 9:19 PM UTC

noisenotsignal wrote: post #6

alexyang21 wrote: post #2

Studies [1] show that a work sample test is the best predictor of candidate performance on the job, which is why many software engineering teams use take-home tests as one step in their hiring process. But designing an effective test is difficult and time-consuming. For example, candidates are reluctant to complete tests that are too long or not engaging enough. But make them too short and teams won’t get the signal…

While this certainly sounds like reasonable criteria for rating a take home, does this truly account for general candidate reluctance to do take homes due to the time commitment? Even 2-4 hours is more time than conventional Leetcode-style initial assessments, and many may rather use that time to interview with 2-4 companies instead of just one. I recently interviewed with Ramp and enjoyed their model of practical pr…

I think most 2-4 hour take-homes can be condensed to 1-2 hours with some thoughtful choices. A few ideas I've used:

- Provide starter code and setup instructions so candidates don't waste time on boilerplate.

- Abbreviate requirements to what actually matters. E.g. do you really need 100% test coverage on a take-home? Ask candidates to write a few tests and then tell you what else they'd do given more time.

- Use an open-ended, time-boxed format instead of having end-to-end expectations. IMO a hybrid format where a short (1 hr) take-home is followed by a live discussion/pairing afterward can be the core component of a hiring process.

I'd love to hear more about the Ramp process. Do you mind sharing what sort of practical problems they used?

Re: We compiled a library of realistic engineering take-home tests and ranked them

#153

Post by alexyang21 » Thu, Aug 25, 2022, 9:21 PM UTC

q7xvh97o2pDhNrh wrote: post #140

alexyang21 wrote: post #131

Earlier quoted context omitted.

The reason why there aren't any 1-star projects isn't because they don't exist, but because we didn't think anyone would want to see them. Cataloguing these was a ton of work (we sifted through hundreds of tests). Including the 1-star ones seemed like it would only shame companies who used them.

I appreciate that you might not want to name-and-shame. I wonder if you could just scrub the identifying info and post them, though? Personally I think it'd still be tremendously valuable to show clear examples of what "bad" looks like in your ranking system. Not only does it help you show off more of the hard work you did, but it also helps all your readers get a more complete understanding of your framework. Beyond…

You've given me a great idea for a future post - thank you! I'll see if I can do a more detailed teardown of a "great" take-home vs. a "bad" one.

Re: We compiled a library of realistic engineering take-home tests and ranked them

#154

Post by throwaway5959 » Thu, Aug 25, 2022, 9:23 PM UTC

thisiswrongggg wrote: post #151

Unless I'm desperate I'll never do an unpaid take home test again in my life. For me it has been a waste of time and hugely demoralizing. Best/worst story and the one that broke camel's back: at one uber-copycat company I had my test failing on one reviewer. Then after a few months another reviewer interviewing me for another company told me that "this test would be fine by me. Had I reviewed it when I was back at th…

What’s wrong with SOLID?

Re: We compiled a library of realistic engineering take-home tests and ranked them

#155

Post by alexyang21 » Thu, Aug 25, 2022, 9:28 PM UTC

francisofascii wrote: post #81

I like the assignments where a starting project is provided, and you are given a few enhancements and bug fix tasks within the project. That is probably what your first few months will be like at the job anyway. You can match the coding standard of the existing project rather than agonizing about naming standards or other subjective decisions. I have an issue with the more open ended tasks where you create a project…

I agree! It's great when companies take the time to do this. Though it's tough when it's been a few months/years and the code hasn't been maintained. You'll see some repos in the library that are up to 8 years old.

One of the ideas my team brainstormed is to create and maintain a library like this that engineering teams can rely on.

Re: We compiled a library of realistic engineering take-home tests and ranked them

#156

Post by thisiswrongggg » Thu, Aug 25, 2022, 9:29 PM UTC

alexyang21 wrote: post #2

Studies [1] show that a work sample test is the best predictor of candidate performance on the job, which is why many software engineering teams use take-home tests as one step in their hiring process. But designing an effective test is difficult and time-consuming. For example, candidates are reluctant to complete tests that are too long or not engaging enough. But make them too short and teams won’t get the signal…

From the paper you link:

"This article summarizes the practical and theoretical implications of 85 years of research in personnel selection. On the basis of meta-analytic findings, this article presents the validity of 19 selection procedures for predicting job performance and training performance and the validity of paired combinations of general mental ability (GMA) and the 18 other selection procedures. Overall, the 3 combinations with the highest multivariate validity and utility for job performance were GMA plus a work sample test (mean validity of .63), GMA plus an integrity test (mean validity of .65), and GMA plus a structured interview (mean validity of .63)"

1. The research is dated (1998). Long before many of the current best practices in SW Eng were established.

2. Says it is based on 85 years of research. Obviously not IT-related then.

3. Even if we get past that it gives 3 almost equally good methods of hiring where the highest one is GMA and integrity test - not work sample test.

4. Even if we get past all that work sample means work sample. It does not have to be produced under pressure in a weekend as unpaid work which as any professional knows is very hard to bring yourself to do right (being a professional means getting paid for my services as I live from selling them). It can very well be some past work on github etc.

So, unless there is a better/more focused on IT/up to date research proving that take home tests (which btw can be offshored/gamed very easily) lead to better hires I remain highly skeptical of all that and big fan of whiteboard/pair programming.

Re: We compiled a library of realistic engineering take-home tests and ranked them

#157

Post by anchochilis » Thu, Aug 25, 2022, 9:35 PM UTC

alexyang21 wrote: post #137

anchochilis wrote: post #76

Earlier quoted context omitted.

As a candidate, I vastly prefer a take-home to live whiteboarding or pairing. That background sensation of being observed and evaluated greatly interferes with my ability to concentrate and perform at my best. (I'll note that I have no problem pairing in real-world work scenarios! I love discussing problems with my team. It's the artificial pressure injected by the interview process that really gets to me)

How would a take-home compare to a hybrid format? For example, if you were given a 1-hour take-home followed by a live session asking about your thought process and then pairing to extend what you previously worked on.

I think a hybrid format is the right way to do take-homes. The candidate is less likely to resent the time invested in the take-home if they have a chance to discuss or "show off" their work. It also gives candidates who have less free time to complete a polished solution the chance to say "if I had more time, I would [add tests, optimize this function, ...]>"

I still struggle with the "observer effect" during live extension exercises, but it's less pronounced than when starting with a problem from scratch. If I prepare a high quality solution in advance and do well in technical discussions, that's usually enough to offset any fumbling around during the live programming session due to interview jitters.

Re: We compiled a library of realistic engineering take-home tests and ranked them

#158

Post by GeorgeWBasic » Thu, Aug 25, 2022, 9:36 PM UTC

morelisp wrote: post #112

irf1 wrote: post #107

Earlier quoted context omitted.

Received 5 PRs, awarded the top 3. You can actually see them https://app.algora.io/algora/challenge/pulls/5 We clarified it in our bounty issue that only the best-winning submissions would get the bounty. Other folks have chosen to award all PR submissions regardless of hiring outcome. The format is up to the company hiring. You're making a great point! It's exactly about what we ask from people who don't get hired.…

I mean you do you, but it's a total farce to call this paying people for their time. You're paying people who win your little contest, which is what hiring already is before your startup tries to insert itself as a middleman.

I don't see how it isn't fraud to claim that.

Re: We compiled a library of realistic engineering take-home tests and ranked them

#159

Post by VoodooJuJu » Thu, Aug 25, 2022, 9:37 PM UTC

Take-home tests are incredibly disrespectful. Not only is it an implication my work and experience are illegitimate, but it's a request for me to do work for free. Not happening.

My frame is always "I'm a professional meeting another professional to see if what I'm selling is fit for what they're seeking to buy, and at a fair price." You can imagine my shock when the interviewer's frame is something like "Do this work for free to prove to me you're worthy." Sorry to burst your bubble, but I have nothing to prove, my friend.

I like to put things like this in mob terms because it distills things down to their simple essence. Mobsters are all about essence and simplicity. No Bullshit™. This humbles the nerd. Here:

>You like steak? I deliver steak. I could deliver you steak. At a good price too.

https://www.youtube.com/watch?v=ii5CiYxwuMo

That's it. That's the essence of the ideal interview. Two professionals having a conversation. No Bullshit™. Like literally every other industry. I'm a respectable steak delivery man and you are a respectable potential client. That's the frame. None of this "prove to me you're worthy" nonsense. When I deliver, you pay. If I don't deliver, you don't pay. Simple as.

>Before I hire you as a plumber, I just need you to install a section of piping free of charge, because I don't trust you or believe who you say you are.

Sounds ridiculous, doesn't it? It is. You'll be installing your own pipe with this attitude.

There's already a great deal of asymmetry in the hiring process. You're getting paid to interview me, but I'm not. Let's not make it any more asymmetric than it already is.

I understand interviewing is difficult. A lot of this difficulty stems from the fact that many of us in this business are pretty nerdy, i.e. socially awkward, both interviewers and candidates alike. This makes it difficult to probe someone via conversation to see if they'd be a good fit. But difficulty with socializing is no excuse. If you're a socially awkward hiring manager, you either need a different role or need to improve your social skills. If you're a socially awkward candidate, same thing goes for you - improve your social skills.

If a hiring manager tries to saddle you with unpaid work, you tell him "fuck you, pay me". Gabish? Have some self-respect people. The more candidates that set these managers straight, the more they'll be forced to start conducting normal human interviews.

Re: We compiled a library of realistic engineering take-home tests and ranked them

#160

Post by bogomipz » Thu, Aug 25, 2022, 9:55 PM UTC

alexyang21 wrote: post #127

bogomipz wrote: post #39

Earlier quoted context omitted.

Agreed. That's been exactly my experience on the few I did before I refused to them. The onus is all on you and your only interface is basically a flakey recruiter. There's no guarantee that any will every look at your submission.

As a candidate, it's an awful feeling to spend time only to get ghosted. It's usually not malicious, just disorganization. From a hiring manager's perspective, it's too easy for things to slip through the cracks. When the volume of candidates increases, it's hard to keep track of all the zip file submissions + individual repos while making sure the eng team reviews them all. This is one of the problems my team is hop…

I didn't say or even imply that it was malicious. Regardless, it's disrespectful and unprofessional. I fail to see how adding yet another SaaS tool into the mix of the dozens that companies already use is somehow going to fix what is essentially a cultural problem. Applicant Tracking Systems have already been a thing for a long time now.