Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI killed the tech interview. Now what?

kanenarraway.com

Post reply on HN
Print view View original

631–640 of 651 posts

Re: AI killed the tech interview. Now what?

#631

Post by xnyan »

I was on a technical interview panel last week where some one of the candidates was using a tool that transcribed our call and fed answers to the candidate in real time.

The first problem was the answers were extremely AI. You could instantly tell that everything they said was kind of off strange. I would not be surprised if this is something that can be improved or addressed, however what will be harder to solve is the fact that the candidate was clearly reading answers. You could see their eyes flick up and parse the text the model was feeding them very clearly. It could not be more obvious.

Re: AI killed the tech interview. Now what?

#632

Post by xnyan »

ipunchghosts wrote: post #614
codr7 wrote: post #302

Earlier quoted context omitted.

Part of the problem is there just aren't a lot of people out there who can correctly judge that level of experience, and looking up the spectrum tends to simply look weird.

I agree. Do you have any thoughts on how to mitigate this? After all, its in your best advantage and the companies to hire someone with talents because of the value they can bring.

Unfortunately (at least in my opinion) the way to mitigate it is make contacts, friends, colleagues etc. Meet people, stay in touch with them. In my experience relationships are by far the most important aspect of a career. I'm not naturally someone who forms these relationships easily, but it's by far the most valuable and important thing I've done in terms of the jobs I've had.

Re: AI killed the tech interview. Now what?

#633

Post by genewitch »

delusional wrote: post #315
m11a wrote: post #248

Earlier quoted context omitted.

I dislike pair programming interviews - as they currently exist - because they usually feel like a time-crunched exam. You don't realistically have the freedom to actually think as you would in actual pair programming. i.e. if you wag your tail chasing a bad end for 15 mins, this is a fail in an interview, but it's pretty realistic of real life work and entirely a non-problem. It's probably even good to test for at i…

I did a couple of rounds of this with my manager as the interviewer. Personally I really liked the process, and the feedback I got from the candidates was positive (but then again it always would be). What worked well for me was that I made it very clear to my manager, a man who I trust, that I would not be able to provide him with a boolean pass/fail result. I couldn't provide him any objective measure of their abil…

This is exactly how I interviewed people and what my replies were like. But I was one of many interviewers, so I would only take 15 minutes, which always annoyed my manager, but I never gave a bad "fine by me", so I still did interviews...

Re: AI killed the tech interview. Now what?

#634

Post by dumbledoren »

> Change our interviews to allow AI: We start testing prompting and refactoring rather than testing coding ability. Can someone program LLMs to get the outcomes they want during an interview? This is a huge shift change and would be uncharted territory.

That is the likely outcome. Its not too difficult to incorporate AI into the process of assessing skill: There is always a risk that the AI will hallucinate or make noticeable mistakes, and being able to detect them would be a confirmation of the engineer's knowledge. Even more so in cases of seniors and architects.

Re: AI killed the tech interview. Now what?

#635

Post by typewithrhythm »

bongoman42 wrote: post #623
typewithrhythm wrote: post #497

Earlier quoted context omitted.

HR involvement is unavoidable at big companies; and basics like "years of experience for payband" can cause issues. They fundamentally do not understand the job, but somehow have to ensure its not a biased hiring process.

Yes, and it is kind of necessary when hiring people from outside trusted networks. HR makes sure if people are who they say they are, background checks, and so on. Years of experience and so on are crude filters and should be bypassable by the team/hiring manager if the candidate meets the requirements. I know in large companies this can become political.

The big issue with the implementations I have seen is because HR aims to minimise the number of checks they do, so they can appear efficient.

So most aim to do as much resume filtering as they can first, then short calls, then background checks.

And the filtering and the call are areas where domain knowledge is relevant.

Re: AI killed the tech interview. Now what?

#636

Post by higeorge13 »

theamk wrote: post #588
qudat wrote: post #477

Earlier quoted context omitted.

> In an hour-long interview, I have to be able to state the problem in a way the candidate can understand, within 10 minutes or so. We don't have time for a lecture on the intricacies of voucher calculation and global sales tax law. Proving if they are technically capable of a job seems rather silly. Look at their resume, look at their online works, ask them questions about it. Use probing questions to understand the…

Have you ever hired? No, it's not enough. There are people out there who can talk great talk, and have great resume, but cannot do their actual job for some reason. Maybe they cannot read the code, maybe they cannot write the code, maybe they can write the code but not in the manner that keeps the rest of codebase working... I've had people like that on my team, it was miserable for all of us. It is essential to see…

I have seen people passing leetcodes and other types of intensive interviews with high ranks but then failed miserably at their jobs. I have seen hard working/learning people, having zero idea on specific programming language but know some basic fundamentals become the best engineers. I know lots of people who are great engineers and have become VPs of engineering, staff engineers, architects, etc. who didn't complete the technical assessment successfully. This means that they weren't perfect at the time of the interview but had other skills to succeed in their jobs in various companies. They were hard workers, great learners, versatile, innovative, etc. But that was a past era of interviews when people were not searching for perfection but mostly evaluating soft skills. Nowadays people just search for the perfect candidate. Any error in the technical assessment or leetcode means rejection, any imperfect/doesn't meet my own solution response also means rejection. Nobody checks for or evaluates soft skills, nobody checks whether I have solved similar problems in some past experience. I guess that's the issue with bad fits, not just the lack of technical skills (which I agree that needs to be evaluated as well but not in an extremely hard and strict yes/no format unless you are google and co).

Re: AI killed the tech interview. Now what?

#637

Post by bberenberg »

karaterobot wrote: post #184

The best interview process I've ever been a part of involved pair programming with the person for a couple hours, after doing the tech screening having a phone call with a member of the team. You never failed to know within a few minutes whether the person could do the job, and be a good coworker. This process worked so well, it created the best team, most productive team I've worked on in 20+ years in the industry,…

This seems like a great idea, and similar to my old approach of “we pay you for 10-20 hours of work across a few days to see how you work with us”.

The downside we found was inconsistency. Candidate 1 and 2 get variable difficulty work. How do we decide who did better?

Re: AI killed the tech interview. Now what?

#638

Post by joshstrange »

AI did not kill tech interviews and anyone saying so is probably not interviewing correctly or trying to outsource the work to one of these other companies.

I’ve run interviews in the last year where I told people they were free to use LLMs in the initial coding (which we don’t watch in realtime) but when they are explaining the code to us we ask them to make a change (a tiny additional feature, 2-6 lines of code depending on how they originally solved it). The people who can’t are immediately disqualified (in our heads, obviously we were cordial in the meeting). Just to note, we let them use LLMs for the change too, it doesn’t help the people who are completely relying on the LLM.

You can use LLMs all you want IF, and only if, you understand what it’s outputting and can build on top of that and/or modify the code it wrote.

As for the deepfake video stuff, ehh, I don’t know as I’ve never seen it “in person” (only videos of how it could look) so I’m not sure if I would notice it or not. We have multiple interviews and I’m confident that speed of answers would give away someone using AI or they would say something stupid the AI hallucinated. That confidence comes from talking to LLMs extensively. They can BS for a little bit but then it becomes clear they full of it.

Also LLMs aren’t going to answer questions like “tell me about a project you worked on at a previous company” or “describe a time where you had to fix production issue and how you went about it”. I had many people trip up here and be unable to describe a past project with any level of detail.

Bottom line, I ask a lot of questions a LLM cannot answer well (not on purpose, these questions were written prior to thinking about LLMs/AI and I ask for people to make a minor edit to code “they” just wrote.

Lastly, the interview process is only the first step. What are these people doing after they get hired? Are they continuing to pay for AI tools to help them lie about who they are? it sure seems like their skill, or rather lack thereof, will become readily apparent even if somehow they made it past the interview process (which again I don’t understand unless you have a bad interview process).

Re: AI killed the tech interview. Now what?

#639

Post by chiph »

karaterobot wrote: post #184

The best interview process I've ever been a part of involved pair programming with the person for a couple hours, after doing the tech screening having a phone call with a member of the team. You never failed to know within a few minutes whether the person could do the job, and be a good coworker. This process worked so well, it created the best team, most productive team I've worked on in 20+ years in the industry,…

Was this an in-person pair programming session or remote via shared desktop?

Because the latter is garbage. You can't see the other person or read their body language. It also gives off "proctored test" vibes. Or worse than vibes - where you're expected to clear your room, swing your camera around, and never look away.

Re: AI killed the tech interview. Now what?

#640

Post by LPisGood »

alsobrsp wrote: post #569
LPisGood wrote: post #506

Earlier quoted context omitted.

> Manager level to replace the dumb question manager. His were all Mac terminal eye candy. He did not get the job Huh? Please explain

We hired the guy who said it was a dump question, he became our manager. He then decided to retire and we had to replace him. One of the candidates answered the 5 cli question with terminal eye candy, not functional commands. He was not hired for the job.

Yeah that’s what you said. I was asking you to explain.

It seems ridiculous to classify any terminal command as “eye candy” and what’s the point of asking the question if you want to hire someone that finds it stupid?

Post reply on HN