Fighting Fire with Fire: Scalable Oral Exams

[...]

> Take-home exams are dead. Reverting to pen-and-paper exams in the classroom feels like a regression.

Yeah, not sure the conclusion of the article really matches the data.

Students were invited to talk to an AI. They did so, and having done so they expressed a clear preference for written exams - which can be taken under exam conditions to prevent cheating, something universities have hundreds of years of experience doing.

I know some universities started using the square wheel of online assessment during covid and I can see how this octagonal wheel seems good if you've only ever seen a square wheel. But they'd be even better off with a circular wheel, which really doesn't need re-inventing.

> The grading was stricter than my own default. That's not a bug. Students will be evaluated outside the university, and the world is not known for grade inflation.

Good!

> 83% of students found the oral exam framework more stressful than a written exam.

That's alright -- that's how life goes. This reminds me of a history teacher I had in middle school who told us how oral exams were done at the university he had studied in: in class, each student would come up to the front, pick three topics at random from a lottery-ball-picker type setup, and then they'd have a few minutes in which to explain how all three are related. I would think that would be stressful except to those who enjoy the topic (in this case: history) and mastered the material.

> Accessibility defaults. Offer practice runs, allow extra time, and provide alternatives when voice interaction creates unnecessary barriers.

Yes, obviously this won't work for deaf students. But why must it be an oral examination anyways? In the real world (see above example) you can't cheat at an oral examination because you're physically present, with no cheat sheets, just you, and you have to answer in real time. But these are "take-at-home" oral exams, so they had to add a requirement of audio/video recording to restore the value of the "physically present" part of old-school oral exams -- if you could do something like that for written exams, surely you would?

Clearly a take-home written exam would be prone to cheating even with a real-time AI examiner, but the real-time requirement might be good enough in many cases, and probably always for in-class exams.

Oh, that brings me to: TFA does not explicitly say it, but it strongly implies that these oral exams were take-at-home exams! This is a very important detail. Obviously the students couldn't do concurrent oral exams in class, not unless they were all wearing high quality headsets (and even then). The exams could have been in school facilities with one student present at a time, but that would have taken a lot of time and would not have required that the student provide webcam+audio recordings -- the school would have performed those recordings themselves.

My bottom-line take: you can have a per-student AI examiner, and this is more important than the exam being oral, as long as you can prevent cheating where the exam is not oral.

PS: A sample of FakeFoster would have been nice. I found videos online of Foster Provost speaking, but it's hard to tell from those how intimidating FakeFoster might have been.

And they didn't even bother to test the most important thing. Were the LLM evaluations even accurate! Have graders manually evaluate them and see if the LLMs were even close or were wildly off.

This is clearly someone who had a conclusion to promote regardless of what the data was going to show.