We compiled a library of realistic engineering take-home tests and ranked them • Hacker News Board

Re: We compiled a library of realistic engineering take-home tests and ranked them

#161

Post by noisenotsignal » Thu, Aug 25, 2022, 9:59 PM UTC

mellavora wrote: post #29

noisenotsignal wrote: post #6

Earlier quoted context omitted.

While this certainly sounds like reasonable criteria for rating a take home, does this truly account for general candidate reluctance to do take homes due to the time commitment? Even 2-4 hours is more time than conventional Leetcode-style initial assessments, and many may rather use that time to interview with 2-4 companies instead of just one. I recently interviewed with Ramp and enjoyed their model of practical pr…

> Even 2-4 hours is more time than conventional Leetcode-style initial assessments, Hard disagree on this one. Unless your day-to-day work includes many leetcode style problems, you need to put in significant time training on leetcode if you want to pass the interview. You should be able to complete a take-home based on your current skill set. Yes, it may take 4-6 hours (or 8-10), and yes, that is a big ask of a cand…

I hard disagree with your disagree :)

1. Some candidates only need to brush up to be in interview shape.

2. To mitigate the need to spend time to prepare, simply use initial assessments to get back in shape. That is, apply to and interview at companies you don’t want to work at first to get more practice problems. If you pass, you get the added benefit of more leverage when negotiating your compensation!

3. As another commenter mentioned, as Leetcode style interviews are shorter, there should be a threshold where $prep_time + $leetcode_interview_time <= $take_home_interview_time. Of course, the threshold adjusts up and down based on your initial level of preparedness.

Re: We compiled a library of realistic engineering take-home tests and ranked them

#162

Post by ghaff » Thu, Aug 25, 2022, 10:04 PM UTC

q7xvh97o2pDhNrh wrote: post #138

yamtaddle wrote: post #136

Earlier quoted context omitted.

> ADDED: I suppose one could argue that the Bar exam is a bit like that but that's a credential as are degrees which are not necessarily all that overlapping with the real world. Yeah, I think most other professions where one might imagine them doing something like a stereotypical software interview, use that instead: a credential obtained by a (perhaps very difficult!) test, maybe with periodic re-tests or required…

> I have a suspicion that a big part of why top-comp software companies keep their interviews so incredibly unpleasant has more to do with discouraging job-hopping among them (so, suppressing wages) than with its being the best process for hiring good developers. You have to wonder if this has backfired. ;) Plenty of people job-hop every 1-2 years, and it's pretty common knowledge which companies ask which level of L…

It's probably pretty common within a certain cluster. A bunch of other people probably see the barriers and just pass. Or my experience is that others just find grass is distinctly not greener and move on.

Re: We compiled a library of realistic engineering take-home tests and ranked them

#163

Post by ghaff » Thu, Aug 25, 2022, 10:06 PM UTC

thaumasiotes wrote: post #116

ghaff wrote: post #30

Earlier quoted context omitted.

It doesn't need to be proctored. It would be easy to start a remote clock and the solution needs to be in by x minutes after the start. It can even be pretty generous. The idea is presumably not to see how fast they can sprint but just put some time-boxing in place so some people aren't taking days. People can get help of course but that's going to be the case absent effectively a remote proctored assignment. And now…

> People can get help of course but that's going to be the case absent effectively a remote proctored assignment. And now you're back to effectively in-person. I would vastly prefer an "in-person" interview in which I wasn't expected to speak to the interviewer to the in-person interviews we actually have. There is a world of difference.

You don't want to ask questions?

Re: We compiled a library of realistic engineering take-home tests and ranked them

#164

Post by alexyang21 » Thu, Aug 25, 2022, 10:19 PM UTC

mr_gibbins wrote: post #67

Picked one at random, the Zero5 assessment, technologies: SQL. The first question was a scenario where I would build a backend endpoint using an API/Lambda and a web frontend. Nothing to do with SQL except for the 'database' icon in the architecture diagram. Rated 5 stars.

Sorry, looks like we didn't tag this one sufficiently! We're updating this now. There was a TON of content to filter through - we're doing our best :)

I also think there's a surprising number of edge cases that need to be considered for this challenge. Consider cars that were already parked at the start of the time range, ones that were entered during the range and never left, etc. So I think it does require familiarity with SQL.

Did you disagree with the tagging, the stars, or both?

Re: We compiled a library of realistic engineering take-home tests and ranked them

#165

Post by speby » Thu, Aug 25, 2022, 10:20 PM UTC

I have a better idea. Don't do take-home assignments as a candidate and don't give them as part of your interview process as an employer. Problem solved.

Re: We compiled a library of realistic engineering take-home tests and ranked them

#166

Post by noisenotsignal » Thu, Aug 25, 2022, 10:25 PM UTC

alexyang21 wrote: post #152

noisenotsignal wrote: post #6

Earlier quoted context omitted.

While this certainly sounds like reasonable criteria for rating a take home, does this truly account for general candidate reluctance to do take homes due to the time commitment? Even 2-4 hours is more time than conventional Leetcode-style initial assessments, and many may rather use that time to interview with 2-4 companies instead of just one. I recently interviewed with Ramp and enjoyed their model of practical pr…

I think most 2-4 hour take-homes can be condensed to 1-2 hours with some thoughtful choices. A few ideas I've used: - Provide starter code and setup instructions so candidates don't waste time on boilerplate. - Abbreviate requirements to what actually matters. E.g. do you really need 100% test coverage on a take-home? Ask candidates to write a few tests and then tell you what else they'd do given more time. - Use an…

I interviewed for a backend role and the problems were greatly simplified versions of day-to-day backend SWE work. For example, use a server to complete a task. The focus was on how you make the requests and handle any edge cases that might come up, and the server was actually live so you could tinker and get immediate feedback. Hopefully that’s not too vague :)

Edit: to be clear, the interview was live with an interviewer. So it wasn’t a take home in the scheduling sense either.

I agree that take homes can be simplified with your suggestions above, and that certainly makes a better experience for the candidate. The hybrid format is also great - future interviews become an extension of your previous work, so it’s more comfortable than having to context switch for a new challenging problem each round.

I didn’t find this in your linked database, but I also enjoyed GitHub’s take home. I only recall spending 45-90 minutes on it, and the setup process was seamless. A recent blog post describes their approach here: https://github.blog/2022-03-31-how-github-does-take-home-tec...

Re: We compiled a library of realistic engineering take-home tests and ranked them

#167

Post by alexyang21 » Thu, Aug 25, 2022, 10:32 PM UTC

llimllib wrote: post #16

All of ours[1] are given three stars but I would love to know why! I feel like they meet your criteria, but who knows. Would love yours, or anybody else's feedback on our problems. We do care about them, and wrote about why we use homework here: https://adhoc.team/2018/02/26/why-we-use-homework-to-recruit... [1]: https://homework.adhoc.team/

Hey Bill! Rating aside, I'm a big fan of Ad Hoc's homework assignments. Here's what I love about them: 1. Your team has made them public for others to use. This is HUGE.

2. The library is organized and the prompts are generally clear and explicit about the requirements to complete.

3. You offer support for candidates with questions about the assignment.

4. Kathy Keating told me about the grading environment you all use with rubrics, blind grading, and rotating cohorts of graders. We actually created similar tooling for the teams we work with.

I also want to explain why they're rated as 3 stars (and why I think this undersells how good they are). When we designed the rating criteria, it was really important to recognize tests that could extract hiring signal without requiring a ton of time from candidates, since that's one of the biggest issues that candidates face. So one of our criteria was "setting clear expectations for candidates (e.g. time expectations)" and another was that the time requested was "reasonable...(Unfortunately, the Ad Hoc tests don't specify time anywhere (technically missing both of these criteria) while doing many other things well that aren't captured in our rubric (e.g. candidate chat tool, blind grading). I admit that our rubric isn't perfect and it feels like the Ad Hoc tests are being doubly penalized for something that's easy to fix. In fact, if you add this to the tests, I'd happily update these to 5-stars.

Finally, if you're up for a chat sometime, I'd love to meet you. I really appreciate the work that your team has done to improve the hiring experience for candidates beyond those at Ad Hoc! You can reach me at alex@trytapioca.com

Re: We compiled a library of realistic engineering take-home tests and ranked them

#168

Post by alexyang21 » Thu, Aug 25, 2022, 10:38 PM UTC

noisenotsignal wrote: post #166

alexyang21 wrote: post #152

Earlier quoted context omitted.

I think most 2-4 hour take-homes can be condensed to 1-2 hours with some thoughtful choices. A few ideas I've used: - Provide starter code and setup instructions so candidates don't waste time on boilerplate. - Abbreviate requirements to what actually matters. E.g. do you really need 100% test coverage on a take-home? Ask candidates to write a few tests and then tell you what else they'd do given more time. - Use an…

I interviewed for a backend role and the problems were greatly simplified versions of day-to-day backend SWE work. For example, use a server to complete a task. The focus was on how you make the requests and handle any edge cases that might come up, and the server was actually live so you could tinker and get immediate feedback. Hopefully that’s not too vague :) Edit: to be clear, the interview was live with an inter…

Yes! I spoke to Andy (the author of the article you linked) when he posted this. I'm a big fan of this approach. And the software we're building is quite similar to the interview-bot one that GitHub uses internally. Why shouldn't every eng team be able to benefit from tools like theirs?

I'd love to add GitHub's take-home to the library, but I feel the article describes the exercise without sharing the actual prompt. If there's a public link to it, lmk and I'll add it.

Re: We compiled a library of realistic engineering take-home tests and ranked them

#169

Post by noisenotsignal » Thu, Aug 25, 2022, 10:54 PM UTC

alexyang21 wrote: post #168

noisenotsignal wrote: post #166

Earlier quoted context omitted.

I interviewed for a backend role and the problems were greatly simplified versions of day-to-day backend SWE work. For example, use a server to complete a task. The focus was on how you make the requests and handle any edge cases that might come up, and the server was actually live so you could tinker and get immediate feedback. Hopefully that’s not too vague :) Edit: to be clear, the interview was live with an inter…

Yes! I spoke to Andy (the author of the article you linked) when he posted this. I'm a big fan of this approach. And the software we're building is quite similar to the interview-bot one that GitHub uses internally. Why shouldn't every eng team be able to benefit from tools like theirs? I'd love to add GitHub's take-home to the library, but I feel the article describes the exercise without sharing the actual prompt.…

You’re right, I don’t think their exercise is public. I just wasn’t sure if you knew about it and felt it was good enough to call out.

This is awesome to hear! I did check out your website but wasn’t sure how closely it tracked with the GitHub method.

Honestly, I had a thought to build something like this after my GitHub interview but wasn’t sure how much traction it would get. It’s cool that you’re working on it and that my random thought was validated; best of luck!

Re: We compiled a library of realistic engineering take-home tests and ranked them

#170

Post by irf1 » Thu, Aug 25, 2022, 11:06 PM UTC

morelisp wrote: post #144

irf1 wrote: post #124

Earlier quoted context omitted.

I'd call it an early experiment and what we could afford at the time to evaluate candidates for a SWE summer internship. Maybe other companies will like this approach and offer challenges that pay much better, with different setups & time horizons. Ex a user turnt his challenge into a 3-month contract-to-hire arrangement that paid $15k to the person who did the work & got the job. Ultimately it's about getting the ri…

> what we could afford at the time to evaluate candidates for a SWE summer internship You didn't even hire them as FT? Christ, what a submarine charade this entire thread has been.

submarine charade, that's pretty funny lol.

The principle of engineers evaluating engineers based on real work remains the same, whether it's for an internship, flexwork, junior or senior roles, no?