Fighting Fire with Fire: Scalable Oral Exams • Hacker News Board

Re: Fighting Fire with Fire: Scalable Oral Exams

#171

Post by cryptonector » Sat, Jan 03, 2026, 1:13 AM UTC

jimbokun wrote: post #85

cryptonector wrote: post #65

Earlier quoted context omitted.

TFA's case involved examinations about the student's submitted project work. It's not the same thing. Even for a more traditional examination with no such context attached one might still want to rely on AI for grading. (Yeah, I know, that comes across as "the students are not allowed to use AI for cheating, but the profs are!".) Also, IMO oral examinations are quite powerful for detecting who is prepared and who isn…

You could argue that for fields like law, medicine and management extroversion and confidence are important qualities.

Quite.

Re: Fighting Fire with Fire: Scalable Oral Exams

#172

Post by sarchertech » Sat, Jan 03, 2026, 1:22 AM UTC

makeitdouble wrote: post #161

sarchertech wrote: post #152

Earlier quoted context omitted.

Very different. A scantron machine is deterministic and non-chaotic. In addition to being non-deterministic LLMs can product vastly different output from very slightly different input. That’s ignoring how vulnerable LLMs are to prompt injection, and if this becomes common enough that exams aren’t thoroughly vetted by humans, I expect prompt attacks to become common. Also if this is about avoiding in person exams, wha…

I saw this piece as the start of an experiment, and the use of a "council of AI" as they put it to average out the variability sounds like a decent path to standardization to me (prompt injecting would not be impossible, but getting something past all the steps sounds like a pretty tough challenge) They mention getting 100% agreement between the LLMs on some questions and lower rates on other, so if an exam was compo…

> Nothing indeed.

So the whole thing is a complete waste of time then as an evaluation exercise.

>council of AIs

This only works if the errors and idiosyncrasies of different models are independent, which isn’t likely to be the case.

>100% agreement

When different models independently graded tests 0% of grades matched exactly and the average disagreement was huge.

They only reached convergence on some questions when they allowed the AIs to deliberate. This is essentially just context poisoning.

1 model incorrectly grading a question will make the other models more likely to incorrectly grade that question.

If you don’t let models see each other’s assessments, all it takes is one person writing an answer in a slightly different way that causes disagreement among models to vastly alter the overall scores by tossing out a question.

This is not even close to something you want to use to make consequential decisions.

Re: Fighting Fire with Fire: Scalable Oral Exams

#173

Post by andrepd » Sat, Jan 03, 2026, 1:23 AM UTC

recursivecaveat wrote: post #151

NewsaHackO wrote: post #71

Earlier quoted context omitted.

The issue is that it is not scalable, unless there is some dependable, automated way to convert handwriting to text.

It is literally perfect linear scaling. For every student you must expend constant minutes of TA time grading the exam. Why is it unconscionable that the university should have an expense scale at the same rate it receives tuition revenue? $90,000 of tuition pays for a lot of grading hours. I feel that scalability is a cultural meme that has lost the plot.

There are phrases that hn loves and "scalable" is one of them. Here, it is particularly inappropriate.

Some people dream that technology (preferably duly packaged by for-profit SV concerns) can and will eventually solve each and every problem in the world; unfortunately what education boils down to is good, old-fashioned teaching. By teachers. Nothing whatsoever replaces a good, talented, and attentive teacher, all the technologies in the world, from planetariums to manim, can only augment a good teacher.

Grading students with LLMs is already tone-deaf, but presenting this trainwreck of a result and framing it as any sort of success... Let's just say it reeks of 2025.

Re: Fighting Fire with Fire: Scalable Oral Exams

#174

Post by andrepd » Sat, Jan 03, 2026, 1:24 AM UTC

vkou wrote: post #118

NewsaHackO wrote: post #71

Earlier quoted context omitted.

The issue is that it is not scalable, unless there is some dependable, automated way to convert handwriting to text.

I assure you, oral exams are completely scalable. But it does require most of a university's budget to go towards labs and faculty, and not administration and sports arenas and social services and vanity projects and three-star dorms.

> sports arenas and social services and vanity projects and three-star dorms

One of these is not like the others.

Re: Fighting Fire with Fire: Scalable Oral Exams

#175

Post by andrepd » Sat, Jan 03, 2026, 1:35 AM UTC

There are phrases that hn loves and "scalable" is one of them. Here, it is particularly inappropriate.

Some people dream that technology (preferably duly packaged by for-profit SV concerns) can and will eventually solve each and every problem in the world; unfortunately what education boils down to is good, old-fashioned teaching. By teachers. Nothing whatsoever replaces a good, talented, and attentive teacher, all the technologies in the world, from planetariums to manim, can only augment a good teacher.

Grading students with LLMs is already tone-deaf, but presenting this trainwreck of a result and framing it as any sort of success... Let's just say it reeks of 2025.

Re: Fighting Fire with Fire: Scalable Oral Exams

#176

Post by sershe » Sat, Jan 03, 2026, 1:39 AM UTC

Not sure how scalable this is but a similar format was popular in Russia when I went to college long before AI. Typically in a large group with 2-5 examiners; everyone gets a slip with problems or theory questions with enough variation between people, and works on it. You're still not supposed to cheat, but it's more relaxed because of the next part, and some professors would say they don't even care if people copied as long as they can handle part 2.

Part 2 is that when you are ready, an examiner sits with you, looks over your stuff and asks questions about it, like clarifications, errors to see if you can fix them, fake errors to see if you can defend your solution, sometimes even variations or unrelated questions if they are on the fence as to the grade. Typically that takes 3-10 minutes per person.

Works great to catch cheating between students, textbook copying and such.

Given that people finish asynchronously you don't need that many examiners.

As to being more stressful for students I never understood this argument. So is real life.. being free from challenge based stress is for kindergarteners

Re: Fighting Fire with Fire: Scalable Oral Exams

#177

Post by andrepd » Sat, Jan 03, 2026, 1:41 AM UTC

eaglefield wrote: post #29

At the price per student it probably makes sense to run some voluntary trial exams during the semester. This would give students a chance to get acquainted to the format, help them check their understanding and if the voice is very intimidating allow them to get used to that as well. As an aside, I'm surprised oral exams aren't possible at 36 students. I feel like I've taken plenty of courses with more participants a…

Of course they are possible! But it would take a fraction of a day's tuition to pay for a TA to do it, so they want to make a god damn chatbot to do it... Good lord.

They're even more possible if you do an oral exam only on the highest grades. That's the purpose, isn't it? To see if a good, very good, or excellent student actually knows what they're talking about. You can't spare 10 minutes to talk to each student scoring over 80% or something? Please

Re: Fighting Fire with Fire: Scalable Oral Exams

#178

Post by Wowfunhappy » Sat, Jan 03, 2026, 1:43 AM UTC

mrguyorama wrote: post #107

Wowfunhappy wrote: post #72

Earlier quoted context omitted.

I basically agree with the thrust of what you're saying, but also: > I wrote assembly and C++ code by hand, and it was expected to compile. No, I never got a chance to try to compile it myself before submitting it for grading. Do you, like, really think this is the best way to assess someone's ability? Can't we find a place between the two extremes? Personally, I'd go with a school-provided computer with a developmen…

The safe middle space still does not involve a computer Lots of my tests involved writing pseudocode, or "Just write something that looks like C or Java". Don't miss the semicolon at the end of the line, but if you write "System.print()" rather than "System.out.printLn()" you might lose a single point. Maybe. If there were specific functions you need to call, it would have a man page or similar on the test itself, or…

Maybe the middle space doesn't involve a compiler, but I really think computers should be allowed on tests, for a different reason: the computer makes it possible to write out of order. You can go back and add to the beginning without erasing and rewriting everything.

This applies to prose as much as code. A computer completely changes the experience of writing, for the better.

Yes, obviously people made do with analog writing for hundreds of years, yadda yadda, I still think it's a stupid restriction.

Re: Fighting Fire with Fire: Scalable Oral Exams

#179

Post by aqme28 » Sat, Jan 03, 2026, 1:59 AM UTC

Too much focus on what is "scalable." Universities are richer than ever. Just pay teachers to give the oral exams rather than trying to do it for cheap like this.

In my graduate studies in Germany, most of my courses used oral exams. It's fine, and it's battle-tested.

Re: Fighting Fire with Fire: Scalable Oral Exams

#180

Post by gwern » Sat, Jan 03, 2026, 2:01 AM UTC

NewsaHackO wrote: post #71

michaelt wrote: post #62

> We surveyed students before releasing grades to capture their experience. [...] Only 13% preferred the AI oral format. 57% wanted traditional written exams. [...] 83% of students found the oral exam framework more stressful than a written exam. [...] > Take-home exams are dead. Reverting to pen-and-paper exams in the classroom feels like a regression. Yeah, not sure the conclusion of the article really matches the…

The issue is that it is not scalable, unless there is some dependable, automated way to convert handwriting to text.

To clarify the point here for people who didn't read OP: the oral exams here are customized and tailored to the student's individual unique project, that's the point and why they are not written:

> In our new "AI/ML Product Management" class, the "pre-case" submissions (short assignments meant to prepare students for class discussion) were looking suspiciously good. Not "strong student" good. More like "this reads like a McKinsey memo that went through three rounds of editing," good...Many students who had submitted thoughtful, well-structured work could not explain basic choices in their own submission after two follow-up questions. Some could not participate at all...Oral exams are a natural response. They force real-time reasoning, application to novel prompts, and defense of actual decisions. The problem? Oral exams are a logistical nightmare. You cannot run them for a large class without turning the final exam period into a month-long hostage situation.

Written exams do not do the same thing. You can't say 'just do a written exam'. So sure, the students may prefer them, but so what? That's apples and oranges.