Fighting Fire with Fire: Scalable Oral Exams • Hacker News Board

Re: Fighting Fire with Fire: Scalable Oral Exams

#161

Post by makeitdouble » Sat, Jan 03, 2026, 12:28 AM UTC

sarchertech wrote: post #152

makeitdouble wrote: post #145

Earlier quoted context omitted.

How different is it in essence from checking boxes to be scanned by a machine and auto-evaluated to get a one dimention numerical score ? Have exams ever been about humanity and the optics of it ?

Very different. A scantron machine is deterministic and non-chaotic. In addition to being non-deterministic LLMs can product vastly different output from very slightly different input. That’s ignoring how vulnerable LLMs are to prompt injection, and if this becomes common enough that exams aren’t thoroughly vetted by humans, I expect prompt attacks to become common. Also if this is about avoiding in person exams, wha…

I saw this piece as the start of an experiment, and the use of a "council of AI" as they put it to average out the variability sounds like a decent path to standardization to me (prompt injecting would not be impossible, but getting something past all the steps sounds like a pretty tough challenge)

They mention getting 100% agreement between the LLMs on some questions and lower rates on other, so if an exam was composed of only questions where there is near 100% convergence, we'd be pretty close to a stable state.

I agree it would be reassuring to have a human somewhere in the loop, or perhaps allow the students to appeal the evaluation (at cost?) if they is evidence of a disconnect between the exam and the other criteria. But depending on how the questions and format is tweaked we could IMHO end up with something reliable for very basic assessments.

PS:

> Also if this is about avoiding in person exams, what prevents students from just letting their AI talk to test AI.

Nothing indeed. The arms race hasn't started here, and will keep going IMO.

Re: Fighting Fire with Fire: Scalable Oral Exams

#162

Post by acbart » Sat, Jan 03, 2026, 12:40 AM UTC

xmddmx wrote: post #91

acbart wrote: post #42

I have a lot of complicated feelings and thoughts about this, but one thing that immediately jumps to my mind: was the IRB (Institutional Review Board) consulted on this experiment? If so, I would love to know more details about the protocol used. If not, then yikes!

Turns out that under the USA Code of Federal Regulations, there's a pretty big exemption to IRB for research on pedagogy: CFR 46.104 (Exempt Research): 46.104.d.1 "Research, conducted in established or commonly accepted educational settings, that specifically involves normal educational practices that are not likely to adversely impact students' opportunity to learn required educational content or the assessment of e…

I'm afraid you misunderstand what it means to be "exempt" under the IRB. It doesn't mean "you don't have to talk to the IRB", it means "there's a little less oversight but you still need to file all the paperwork". Here's one university's explanation[1]:

> Exempt human subjects research is a specific sub-set of “research involving human subjects” that does not require ongoing IRB oversight. Research can qualify for an exemption if it is no more than minimal risk and all of the research procedures fit within one or more of the exemption categories in the federal IRB regulations. *Studies that qualify for exemption must be submitted to the IRB for review before starting the research. Pursuant to NU policy, investigators do not make their own determination as to whether a research study qualifies for an exemption — the IRB issues exemption determinations.* There is not a separate IRB application form for studies that could qualify for exemption – the appropriate protocol template for human subjects research should be filled out and submitted to the IRB in the eIRB+ system.

Most of my research is in CS Education, and I have often been able to get my studies under the Exempt status. This makes my life easier, but it's still a long arduous paperwork process. Often there are a few rounds to get the protocol right. I usually have to plan studies a whole semester in advance. The IRB does NOT like it when you decide, "Hey I just realized I collected a bunch of data, I wonder what I can do with it?" They want you to have a plan going in.

[1] https://irb.northwestern.edu/submitting-to-the-irb/types-of-...

Re: Fighting Fire with Fire: Scalable Oral Exams

#163

Post by michaelt » Sat, Jan 03, 2026, 12:43 AM UTC

musicale wrote: post #124

perching_aix wrote: post #77

Earlier quoted context omitted.

> Cohorts for programs with a thousand initial students had less than 10 graduates. This was the norm. And why is this a flex exactly? Almost sounds like fraud. Get sold on how you'll be taught well and become successful. Pay. Then be sent through an experience that filters so severely, only 1% of people pass. Receive 100% of the blame when you inevitably fail. Repeat for the other 990 students. The "university thank…

If teaching is the goal, a 99% failure rate seems counterproductive.

I'd wager the "Cohorts for programs with a thousand initial students had less than 10 graduates" statement is deceptive, if not outright false.

Perhaps lifetimerubyist means "1000 students took the mandatory philosophy and ethics 101 class, but only 10 graduated as philosophy majors"

Re: Fighting Fire with Fire: Scalable Oral Exams

#164

Post by jmye » Sat, Jan 03, 2026, 12:44 AM UTC

michaelt wrote: post #142

WJW wrote: post #136

Earlier quoted context omitted.

Regular exams definitely take more than a single hour though. How is this bad?

Talking to inanimate objects is for 5-year-olds and the mentally ill.

What on earth does that have to do with the comment you responded to?

Re: Fighting Fire with Fire: Scalable Oral Exams

#165

Post by grugagag » Sat, Jan 03, 2026, 12:47 AM UTC

knallfrosch wrote: post #134

YakBizzarro wrote: post #11

I seriously don't get it. At my time in university, ALL the exams were oral. And most had one or two written parts before (one even three, the professor called it written-for-the-oral). Sure, the orals took two days for the big exams at the beginning, still, professors and their assistants managed to offer six sessions per year.

Professors are just humans. If they can grade you with an AI for $5 and spend the 20 hours gained scrolling on their phone – guess what, they'll do that.

How about they spend that time preparing to become better teachers/professors? Also there’s a lot of paperwork that eats into their time and energy, why not use AI use AI as a tool to assist?

Re: Fighting Fire with Fire: Scalable Oral Exams

#166

Post by sn9 » Sat, Jan 03, 2026, 12:48 AM UTC

jmye wrote: post #89

perching_aix wrote: post #77

Earlier quoted context omitted.

> Cohorts for programs with a thousand initial students had less than 10 graduates. This was the norm. And why is this a flex exactly? Almost sounds like fraud. Get sold on how you'll be taught well and become successful. Pay. Then be sent through an experience that filters so severely, only 1% of people pass. Receive 100% of the blame when you inevitably fail. Repeat for the other 990 students. The "university thank…

> And why is this a flex exactly? Almost sounds like fraud. Do you think you're just purchasing a diploma? Or do you think you're purchasing the opportunity to gain an education and potential certification that you received said education? It's entirely possible that the University stunk at teaching 99% of it's students (about as equally possible that 99% of the students stunk at learning), but "fraud" is absolute no…

If you have a You could easily raise the bar without sacrificing quality of education (and likely you'd improve it just from the improvement in student:teacher ratio).

Re: Fighting Fire with Fire: Scalable Oral Exams

#167

Post by jmye » Sat, Jan 03, 2026, 12:56 AM UTC

perching_aix wrote: post #135

jmye wrote: post #89

Earlier quoted context omitted.

> And why is this a flex exactly? Almost sounds like fraud. Do you think you're just purchasing a diploma? Or do you think you're purchasing the opportunity to gain an education and potential certification that you received said education? It's entirely possible that the University stunk at teaching 99% of it's students (about as equally possible that 99% of the students stunk at learning), but "fraud" is absolute no…

I don't think one applies to university expecting they're purchasing themselves a diploma, nor that they should be magically absolved of putting in effort to learn the material. What I do think is that the place they describe sounds an awful lot like people being set up for failure though, and so that begged the question as to why that might be. I should probably clarify that I wasn't particularly serious about my fr…

> If teaching was so simple that you could just tell people to go RTFM then recite it from memory, I don't know why people are bothering with pedagogy at all. It'd seem that there's more to teaching and learning than the bare minimum, and that both parties are culpable. Doesn't sound like you disagree on that either.

I do not! A situation where roughly 1% of the class is passing suggests that some part of the student group is failing, and also that there is likely a class design issue or a failure to appropriately vet incoming students for preparedness (among, probably, numerous other things I'm not smart enough to come up with).

And I did take issue with the "fraud" framing; apologies for not catching your tone! I think there is a chronic issue of students thinking they deserve good grades, or deserve a diploma simply for showing up, in social media and I probably read that into your comment where I shouldn't have.

> Jokes aside though, isn't it a gamble?

Not at all. If you learn the material, you pass and get a diploma. This is no more a gamble than your paycheck. However, I think that also presumes that the university accepts only students it believes are capable of passing it's courses. If you believe universities are over-accepting students (and I think the evidence says they frequently are not, in an effort to look like luxury brands, though I don't have a cite at hand), then I can see thinking the gambling analogy is correct.

Re: Fighting Fire with Fire: Scalable Oral Exams

#168

Post by reincarnate0x14 » Sat, Jan 03, 2026, 1:06 AM UTC

vasco wrote: post #120

michaelt wrote: post #62

> We surveyed students before releasing grades to capture their experience. [...] Only 13% preferred the AI oral format. 57% wanted traditional written exams. [...] 83% of students found the oral exam framework more stressful than a written exam. [...] > Take-home exams are dead. Reverting to pen-and-paper exams in the classroom feels like a regression. Yeah, not sure the conclusion of the article really matches the…

One student had to talk to an AI for more than 60 minutes. These guys are creating a dystopia. Also students will just have an AI pick up the phone if this gets used for more than 2 semesters.

A Fire Upon the Deep coming to your classroom!

Re: Fighting Fire with Fire: Scalable Oral Exams

#169

Post by rogerrogerr » Sat, Jan 03, 2026, 1:09 AM UTC

Levitz wrote: post #90

rogerrogerr wrote: post #46

Earlier quoted context omitted.

> now you could just run all of that student's activity in class through that AI. In the real world you don't know if someone is competent because you run an exam, you know if he is competent because he consistently shows competency. But isn’t the whole point of a class to move from incompetent to competent?

Sure, and the exam is to test that happened. There is no need to perform that test at one point in time if you continuously check the student's performance.

Ah, now I’m getting it. You’re basically measuring the derivative of competency and getting a decent idea of where they are at the end of the course without needing to do a big-bang final exam.

Re: Fighting Fire with Fire: Scalable Oral Exams

#170

Post by Eisenstein » Sat, Jan 03, 2026, 1:11 AM UTC

makeitdouble wrote: post #161

sarchertech wrote: post #152

Earlier quoted context omitted.

Very different. A scantron machine is deterministic and non-chaotic. In addition to being non-deterministic LLMs can product vastly different output from very slightly different input. That’s ignoring how vulnerable LLMs are to prompt injection, and if this becomes common enough that exams aren’t thoroughly vetted by humans, I expect prompt attacks to become common. Also if this is about avoiding in person exams, wha…

I saw this piece as the start of an experiment, and the use of a "council of AI" as they put it to average out the variability sounds like a decent path to standardization to me (prompt injecting would not be impossible, but getting something past all the steps sounds like a pretty tough challenge) They mention getting 100% agreement between the LLMs on some questions and lower rates on other, so if an exam was compo…

A technological solution to a human problem is the appeal we have fallen for too many times these last few decades.

Humans are incredibly good at solving problems, but while one person is solving 'how do we prevent students from cheating' a student is thinking 'how I bypass this limitation preventing me from cheating'. And when these problems are digital and scalable, it only takes one student to solve that problem for every other student to have access to the solution.