Fighting Fire with Fire: Scalable Oral Exams • Hacker News Board

Re: Fighting Fire with Fire: Scalable Oral Exams

#281

Post by viccis » Sun, Jan 04, 2026, 3:45 AM UTC

Panos wrote: post #200

Just in case, I am the author of the blog post. For our "AI" class, it felt like a good class to experiment with something novel. No, we do not want to eliminate the pen and paper exam. It works well. We use it. The oral exam is yet another tool. Not a solution for everything. In our case, we wanted to ensure that the students who worked on the team project: (a) contributed enough to understand the project, (b) actua…

>We do allow them to use LLMs, it would be stupid not to.

I'm not sure why you're saying this so confidently. Using LLMs on school work is like using a forklift at the gym. You'll technically finish the task you set out to do, and it will be much easier. So why not use a forklift at the gym?

>But the key reason was the following: research has shown that human interviewers are actually worse when they get tired, and that AI is actually better for conducting more standardized and more fair interviews. That result was a major reason for us to trust a final exam on a voice agent.

I think that in an "AI class" for MBA students, the material is probably not complex enough to require much more than a Zork interpreter, but if you tried this on something in which nuance is required, that comparison would change dramatically. For something like this, which is likely going to be little more than knowledge spot checks to catch the most blatant cheaters, why not just have students do multiple choice questions at a kiosk?

Re: Fighting Fire with Fire: Scalable Oral Exams

#282

Post by JanisErdmanis » Sun, Jan 04, 2026, 12:59 PM UTC

YakBizzarro wrote: post #11

I seriously don't get it. At my time in university, ALL the exams were oral. And most had one or two written parts before (one even three, the professor called it written-for-the-oral). Sure, the orals took two days for the big exams at the beginning, still, professors and their assistants managed to offer six sessions per year.

When I did my BSc and MSc in physics almost all my exams were oral just like you described. Latter I did a PhD in a different university where oral exams were never practiced. My PhD supervisor told me that part of it is because of the scaling issue, but another very interesting point he made is that it is about cultural interpretation of fairness.

In my BSc and MSc we were all basically locals who are in all aspects about the same except from the aptitude to study. In the university where I did my PhD there were much more divisions (aka diversity) in which every oral examiner would need to navigate so one group does not feel to be made preferential over another.

Re: Fighting Fire with Fire: Scalable Oral Exams

#283

Post by sarchertech » Sun, Jan 04, 2026, 2:54 PM UTC

aaplok wrote: post #193

sarchertech wrote: post #146

Earlier quoted context omitted.

Distance education is a tiny percentage of higher education though. Online classes at a local university are more common, but you can still bring the students in for proctored exams. Even for distance education though, proctored testing centers have been around longer than the internet.

> Distance education is a tiny percentage of higher education though. It is about a third of the students I teach, which amounts to several hundreds per term. It may be niche, but it is not insignificant, and definitely a problem for some of us. > Even for distance education though, proctored testing centers have been around longer than the internet. I don't know how much experience you have with those. Mine is exten…

I agree that proctoring across hundreds of locations globally could be a challenge.

I think the end result though is that schools either limit their students to a smaller number of locations where they can have proctored exams, or they don’t and they effectively lose their credentialing value.

Re: Fighting Fire with Fire: Scalable Oral Exams

#284

Post by itissid » Sun, Jan 04, 2026, 8:59 PM UTC

A colleague of mine raised a very important point here. The class is being taught at NYU business school(co taught Konstantinos Rizakos AI/ML Product Mgmt). The fees is pretty high 60,000/year ($2,000+/credit @15 credits/sem) . How much of an ask is it on the business model to incorporate human evaluation say 25% of the cost ~15000$ to spending per student to have their exams evaluated orally by a TA or just do the damn exam in a controlled class environment?

Re: Fighting Fire with Fire: Scalable Oral Exams

#285

Post by Panos » Sun, Jan 04, 2026, 9:41 PM UTC

viccis wrote: post #281

Panos wrote: post #200

Just in case, I am the author of the blog post. For our "AI" class, it felt like a good class to experiment with something novel. No, we do not want to eliminate the pen and paper exam. It works well. We use it. The oral exam is yet another tool. Not a solution for everything. In our case, we wanted to ensure that the students who worked on the team project: (a) contributed enough to understand the project, (b) actua…

>We do allow them to use LLMs, it would be stupid not to. I'm not sure why you're saying this so confidently. Using LLMs on school work is like using a forklift at the gym. You'll technically finish the task you set out to do, and it will be much easier. So why not use a forklift at the gym? >But the key reason was the following: research has shown that human interviewers are actually worse when they get tired, and t…

I agree that I am not yet confident to use this approach for my technical classes. I am still very unhappy with any option for assessment for technical classes, but I would not trust an LLM to come up with good questions. NotebooksLM does come up with decent quizzes, but nothing super hard.

For the use of LLM in classes: I understand the reasoning, but I found LLMs to be extremely educational for parsing through dense material (eg parsing an NTSB report for an Uber self-driving crash). Prohibiting students from using LLMs would be counterproductive.

But I still want students to use LLMs responsibly, hence the oral exam.

Re: Fighting Fire with Fire: Scalable Oral Exams

#286

Post by Panos » Sun, Jan 04, 2026, 11:00 PM UTC

itissid wrote: post #284

A colleague of mine raised a very important point here. The class is being taught at NYU business school(co taught Konstantinos Rizakos AI/ML Product Mgmt). The fees is pretty high 60,000/year ($2,000+/credit @15 credits/sem) . How much of an ask is it on the business model to incorporate human evaluation say 25% of the cost ~15000$ to spending per student to have their exams evaluated orally by a TA or just do the d…

Not an issue of cost, at all.

Absolutely the easiest solution would have been to have a written exam on the cases and concepts that we discussed in class. It would take a few hours to create and grade the exam.

But at a university you should experiment and learn. What better class to experiment and learn than the “AI Product Management”. Students were actually intrigued by the idea themselves.

The key goal: we wanted to ensure that the projects that students submitted was actually their own work, not “outsourced” (in a general sense) to teammates or to an LLM.

Gemini 3 and NotebookLM with slide generation were released in the middle of the class, and we realized that it is feasible for a student to have a flaweless presentation in front of the class, without understanding deeply what they are presenting.

We could schedule oral exams during the finals week, which would be a major disruption for the students, or schedule exams during the break, violating university rules and ruining students vacation.

But as I said, we learned that AI-driven interviews are more structured and better than human-driven ones, because humans do get tired, and they do have biases based on who is the person they are interviewing. That’s why we decided to experiment with voice AI for running the oral exam.

Re: Fighting Fire with Fire: Scalable Oral Exams

#287

Post by mirrir » Tue, Jan 06, 2026, 3:21 PM UTC

latexr wrote: post #264

mirrir wrote: post #215

My university had a great policy for this. For every major assignment you went through interview grading. if you failed it you lost 60% of that grade.

> interview grading Would you mind expanding on what exactly that entails?

Yes absolutely. For large code writing assignments and projects grad students would be tasked with writing the code. After submission they had to schedule a 15-20 minute chat with a grad student in the course for Interview Grading. Through that process the grad would ask why the student made specific choices in their code, where improvements could have been made, and the process they took to solve the task. It wound up being a pretty effective and kind way to get people to not really care as much about referencing things like StackOverflow (no GPT at the time), and helped make a lot of students need to care much more aabout the why of their code.

Re: Fighting Fire with Fire: Scalable Oral Exams

#288

Post by jasonfarnon » Wed, Jan 07, 2026, 12:45 AM UTC

Traubenfuchs wrote: post #273

jasonfarnon wrote: post #160

Earlier quoted context omitted.

I have been wondering if some of my students who demonstrated zero knowledge in class but ace in-class exams were doing something like this. I figured something like a hacked out google glasses would do the trick.

They probably just have huge pools of all your previous tests that they share and memorize.

No, that wouldn't explain why this has only occurred in the last ~2 years.