Fighting Fire with Fire: Scalable Oral Exams

If you can use AI agents to give exams, what is stopping you from using them to teach the whole course?

Also, with all the progress in video gen, what does recording the webcam really do?

Being interrogated by an AI voice app... I am so grateful I went to university in the before time

If this is the only way to keep the existing approach working, it feels like the only real solution for education is something radically different, perhaps without assessment at all

> Many students who had submitted thoughtful, well-structured work could not explain basic choices in their own submission after two follow-up questions.

When I was doing a lot of hiring we offered the option (don’t roast me, it was an alternative they could choose if they wanted) of a take-home problem they could do on their own. It was reasonably short, like the kind of problem an experienced developer could do in 10-15 minutes and then add some polish, documentation, and submit it in under an hour.

Even though I told candidates that we’d discuss their submission as part of the next step, we would still get candidates submitting solutions that seemed entirely foreign to them a day later. This was on the cusp of LLMs being useful, so I think a lot of solutions were coming from people’s friends or copied from something on the internet without much thought.

Now that LLMs are both useful and well known, the temptation to cheat with them is huge. For various reasons I think students and applicants see using LLMs as not-cheating in the same situations where they wouldn’t feel comfortable copying answers from a friend. The idea is that the LLM is an available tool and therefore they should be able to use it. The obvious problem with that argument is that we’re not testing students or applicants on their abilities to use an LLM, we’re using synthetic problems to explore their own skills and communication.

Even some of the hiring managers I know who went all in on allowing LLMs during interviews are changing course now. The LLM-assisted interviewed were just turning into an exercise of how familiar the candidate was with the LLM being used.

I don’t really agree with some of the techniques they’re using in this article, but the problem they’re facing is very real.

...if I was a student, I just fundamentally don't think I'd want to be tested by an AI. I understand the author's reasoning, but it just doesn't feel respectful for something that is so high-stakes for the student.

Wouldn't a written exam--or even a digital one, taken in class on school-provided machines--be almost as good?

As long as it's not a hundred person class or something, you can also have an oral component taken in small groups.