Interviewing Engineers in the AI Era: Lessons from a Year of Rebuilding

Interviewing Engineers in the AI Era: Lessons from a Year of Rebuilding
coinbase.com

So I'm increasingly uncertain about what and how to test. My default for now is still to rely on ability to write basic code fluently, but I'm open to changing this perspective.

I really want to know how this existing repo AI-assisted live coding test works, with example problems.

It seems the standard data structure puzzle type thing won't be feasible if you are using an LLM.

Also the latency for these agentic coding/prompts seems like it would make the interview a bit awkward.

Anyone been conducting or taking interviews with this kind of thing with thoughts to share?

- You get a leetcode question and if you're lucky is an easy medium that you can solve, if you're really lucky you already solved it and can pretend you are approaching the problem the first time. Good luck if you get a hard question and you never saw it before.

- You get a home assignment, in a framework you might not know but you're expected to be fluent with it, then waste 1 hour setting up the project structure, and one more hour to find out how the framework expects you to define the CORS allow list. You are expected to deliver the project in 3 hours.

The good I see in AI is that it completely removes the need to study just for interviews, and you can also delegate all the project setup to the AI. Then you can focus on what you would test (e2e? integration? what are the boundaries? what do we mock?), how to keep the documentation, how to structure your code. You have an expensive endpoint, do I make it sync or add an async jobs framework?

Imagine you're an expert in C++ interviewing for a Django position and the interview consists of fixing a big in a repo. The bug is that a function without type hinting is modifying what is expected to be a list, but the caller is passing a tuple. Trivial after a week you work in python and you have your environment set up for type warnings, also trivial with AI and definitely not an interesting problem that shows expertise with software engineering in general.

We also did this in our last interview at work, and it was a really good indicator to see if someone just copy pasted code, or understood it after it was generated. Some candidates had a unit test fail and couldn't debug it for his life, even if he "wrote" all the code himself. Others simply did not understand the architecture they wrote, and assumed that a function defined with "async" and awaited would run in parallel from the code that called it (as if you spawned a thread)

They have to earn it, as the tokens are not free.

Given that deskilling and over-reliance in AI assistance will continue to happen, putting a hard token limit Do you want a candidate that knows when to use AI and carefully uses tokens with in their limits, or do you want a candidate generating incomprehensible AI slop to be tokenmaxxing out your company limits and then draining your company bank account?