Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Hiring Post

sockpuppet.org

Post reply on HN
Print view View original

61–70 of 266 posts

Re: The Hiring Post

#61

Post by tptacek »

meowface wrote: post #52
eps wrote: post #43

We went through the phase when we gave candidates a problem and let them work on it remotely. It was in an embedded C shop that did a lot of kernel work. Basically, we'd give a short programming task (say, to write an intrusive AVL tree container in C) and 24 hours. Guess what? HALF of candidates cheated. Meaning that when the got called for an in-person interview, they stumbled to explain how "their" code worked. To…

Yeah, that's the only real problem with remote work sample tests. Thankfully, with application security, you can mitigate most kinds of cheating by presenting a custom-written black box and require a candidate to attack it in some way. Unless previous candidates are leaking or sharing info, there's not that much you can do to cheat on that, especially if you have some reasonable time limit (< 5 hours).

Actually, I think the opposite is true; in appsec, the final work product is a list of vulns, so if you want to cheat, it only takes 4 minutes talking to anyone else who has ever succeeded in the process before.

For a dev interview, taking something your team has built and scooping out some of its functionality seems like a test that is significantly harder to cheat on.

For my part: we ran this process for over two years and never discovered any plagiarism. Meanwhile, we drastically increased the size of our team and had total retention; from the time I took over recruiting to the time I left the firm, we lost a total of zero of those hires. All of them worked out.

Re: The Hiring Post

#62

Post by sytelus »

Work-sample questions is not free from its own evils. They are expensive to device. At big companies, significant portion of candidates dump out their interview questions on Internet so that's a big challenge.

Second, work-sample type of questions are also expensive to answer for candidates so you can probably just ask one question as part of whole interview and that's about it. So any conclusion is drawn from sample of 1 as opposed to sample of 5.

Third, work-sample type of questions requires knowledge of engineering skills that candidate may or may not have developed well at point in time. At many companies, emphasis is how can we solve problem X from algorithms perspective and engineering is just something you learn on the way if you haven't already. Key is ability to figure out computer science part rather than language and tooling part.

Overall, I think many companies blindly try to follow the model of Google, Facebook etc and fall on their face. If you expect your developer to write CRUD apps or do mostly plumbing work, there is no point in asking questions like find common ancestor in binary tree. At other companies, developers are expected to solve computational problems and engineering/plumbing/CRUD is small part of the solution pie. There it makes a LOT of sense to ask candidate to solve problems that requires deep computer science. Also asking as many of as possible so that your sample space is large and outcome has more confidence.

Re: The Hiring Post

#63

Post by bentcorner »

nissimk wrote: post #41
tptacek wrote: post #27

Author. I can talk in a pretty good amount of detail about how exactly our process worked, if anyone has any questions. And, to head off a concern a reviewer gave me: from 1997-2005, I was a full-time software developer; I shipped shrink-wrap boxed software on Windows and Unix in the 1990s, then appliances deployed at tier 1 ISPs. I'm a "software person" more than a "security person".

It's funny, because to change it, you have to admit that the way you were doing it before is not effective. I've come across many people who admit that they are bad at being interviewed, but very few people I meet have admitted to being bad at interviewing others. It's incongruous with the results, because these are the same people that have hired folks that we all agree are not exhibiting the level and characteristi…

I'll admit that I'm bad at interviewing others. I'm not trying to humble-brag or anything. I'm just bad at it. Part of it is that I haven't had to go through an interview in over 10 years, so it's difficult to empathize with a candidate. I do my best, though. The other part is that I'm happy at what I do, but I'm not the best. I sometimes can't keep up with candidates when they're writing on a whiteboard and make a claim that such-and-such works. I squint and can make a judgement on that claim, but really I need to cover x, y, and z during our hour together and we're already 20 minutes into it and I don't want to get dragged into the weeds.

So, I'm not a fan of giving interviews. I wouldn't mind shadowing someone better than I, but my first inclination is that that would be a terrible idea since it would make candidates even more nervous.

Re: The Hiring Post

#64

Post by jowiar »

nvarsj wrote: post #22
jowiar wrote: post #18

This article seems to put forward a good theory on interviewing someone who you want to lock in room with a problem and have code come out the other end. Unfortunately, most of us work in the real world, where that's a very limited portion of the engineering job. When the author says selecting for: "people who have the social skills to actively listen to someone else’s technical points, to guide a discussion with que…

> And, at least in my job, engineers spend a whole lot more time dealing with people than with code. I'd say something is wrong then. If your business depends on shipping code why are your developers talking to people most of the time?

Business depends on solving problems. Code is a medium for implementing solutions. I can count the number of times I've needed to put my CS degree to use on one hand, but knowing to whom, when, how to ask the right questions has been worth far more.

Re: The Hiring Post

#65

Post by meowface »

tptacek wrote: post #61
meowface wrote: post #52

Earlier quoted context omitted.

Yeah, that's the only real problem with remote work sample tests. Thankfully, with application security, you can mitigate most kinds of cheating by presenting a custom-written black box and require a candidate to attack it in some way. Unless previous candidates are leaking or sharing info, there's not that much you can do to cheat on that, especially if you have some reasonable time limit (< 5 hours).

Actually, I think the opposite is true; in appsec, the final work product is a list of vulns, so if you want to cheat, it only takes 4 minutes talking to anyone else who has ever succeeded in the process before. For a dev interview, taking something your team has built and scooping out some of its functionality seems like a test that is significantly harder to cheat on. For my part: we ran this process for over two y…

You're right that it's definitely pretty easy to cheat if you have insider knowledge of some sort, but the odds of a candidate leaking the answers are probably pretty low unless you're a huge company and don't change up your tests.

Re: The Hiring Post

#66

Post by tptacek »

sytelus wrote: post #62

Work-sample questions is not free from its own evils. They are expensive to device. At big companies, significant portion of candidates dump out their interview questions on Internet so that's a big challenge. Second, work-sample type of questions are also expensive to answer for candidates so you can probably just ask one question as part of whole interview and that's about it. So any conclusion is drawn from sample…

We had three work-sample exercises at Matasano. All told, they took candidates mid-single-digit hours to finish (many geeked out, or golfed on them; we did our best to keep this from happening --- you can read about this on our hiring page:

http://www.matasano.com/careers

This sounds onerous, but it is less onerous than the normal dev hiring process, which involves an onsite interview that eats the whole day. We did on-site interviews, but they took just 2.5 hours to complete; candidates would be out at lunch. Essentially, we were shifting some of the time candidates would spend sweating in a conference room to their couch instead.

There are bad work-sample tests. "Bad" comes in a variety of flavors. I think bad work sample tests will out-predict the median interview. But there's a lot of room to optimize and improve this process, sure.

Re: The Hiring Post

#67

Post by smenko »

Great, if you want to hire people who "can code", but have no social skills. I am personally done working in environments where people think they "can code" but have not enough of a social skill to at least empathise with your point of view on naming conventions.

Don't get me wrong, I can code, I'll write you anything you want, AND IT WILL WORK! and I'll still know there may be a better way but this is the way that works and I still think I don't have enough years to live to argue with social incompetents who think that fighting over tabs or spaces is a good idea and a pocket protector makes you sexy.

Re: The Hiring Post

#68

Post by tptacek »

smenko wrote: post #67

Great, if you want to hire people who "can code", but have no social skills. I am personally done working in environments where people think they "can code" but have not enough of a social skill to at least empathise with your point of view on naming conventions. Don't get me wrong, I can code, I'll write you anything you want, AND IT WILL WORK! and I'll still know there may be a better way but this is the way that w…

We are not selecting for "no social skills". We're avoiding selecting for slickness.

Re: The Hiring Post

#69

Post by cyounkins »

tptacek wrote: post #27

Author. I can talk in a pretty good amount of detail about how exactly our process worked, if anyone has any questions. And, to head off a concern a reviewer gave me: from 1997-2005, I was a full-time software developer; I shipped shrink-wrap boxed software on Windows and Unix in the 1990s, then appliances deployed at tier 1 ISPs. I'm a "software person" more than a "security person".

How do you avoid wasting time on candidates that are a poor fit? When we advertise for an open position we get a ton of unqualified candidates.

In your post you say, "At my last firm, we had the “first-call” system. Every serious applicant got, on average, 30-45 minutes of director-level time on the phone before any screening began." You seem to continue on from that point. What happens before first-call?

Re: The Hiring Post

#70

Post by tptacek »

cyounkins wrote: post #69
tptacek wrote: post #27

Author. I can talk in a pretty good amount of detail about how exactly our process worked, if anyone has any questions. And, to head off a concern a reviewer gave me: from 1997-2005, I was a full-time software developer; I shipped shrink-wrap boxed software on Windows and Unix in the 1990s, then appliances deployed at tier 1 ISPs. I'm a "software person" more than a "security person".

How do you avoid wasting time on candidates that are a poor fit? When we advertise for an open position we get a ton of unqualified candidates. In your post you say, "At my last firm, we had the “first-call” system. Every serious applicant got, on average, 30-45 minutes of director-level time on the phone before any screening began." You seem to continue on from that point. What happens before first-call?

To me, recruiting is two problems: outreach and qualification. You're asking about outreach. How we did outreach is a whole 'nother blog post, and one I'm bound to write soon.

What surprised me about recruiting was how much more important qualification is than outreach. Without good qualification, it almost doesn't matter how good your outreach is, because you're filtering for the same highly visible easily accessible candidates every other firm is looking for. The best-priced talent is buried. What you need is ground-penetrating radar. Clever outreach schemes helped us, but what really let us make our best hires was a qualification process that allowed us to confidently ignore resumes.

We didn't just maintain our team's level of quality by ignoring resumes. We transcended it. We never could have run a resume-based process that would hire people who could use lattice math to break crypto. To hire those people, we needed to select for aptitude, not experience.

I haven't said it anywhere yet, but in case the subtext isn't clear: the person who can implement an attack for which edit/compile/debug takes 4-6 hours also does a pretty great job on every other facet of the software security job. There are aptitudes that are very good proxies for other aptitudes!

Post reply on HN