Fighting Fire with Fire: Scalable Oral Exams

LLM oral exams can provide assessment in a student's native language. This can be very important in some scenarios!

Unlimited attempts won't work in the presented model. No matter how many cases you have, all will eventually find their way to the various cheating sites.

There is no silver bullet. There's no solution that works for all schools. Strategies that work well for M.I.T. with competitive enrollment and large budgets won't work for a small community college in an agricultural state, with large teaching loads per professor, no TAs, and about 15-25 hours of committee or other non-teaching work. That was my situation.

Teaching five courses and eight sections, 20-30 students per section, 10-20 office hours every week (and often more if the professor cared about the students), leaves little time for grading. In desperation I turned to weekly homework assignments, 4-6 programming projects, and multiple choice exams (containing code and questions about it). Not ideal by any means, just the best I could do.

So I smile now (I'm retired) when I hear about professors with several TAs each, explaining how they do assessment of 36 students at a school with competitive enrollment.

Yeah, that's fine, I can definitely appreciate that angle too.

As you can probably surmise, I've had quite some struggles during my college years specifically, hence my angle of concern. It used to be the other way around, I was doing very well prior to college, and would always find people's complaints to be just excuses. But then stuff happened, and I was never really the same. The rest followed.

My personal sob story aside, what I've come to find is that while yes, a lot of the things slackers say are cheap excuses or appeals to fringe edge-cases, some are surprisingly valid. For example, if this aforementioned 99% attrition rate is real, that is very very suspect. Worse still though, I'd find things that people weren't talking about, but were even more problematic. I'll have to unfortunately keep that to myself though for privacy reasons [0] [1].

Regarding grading, I find grade inflation very concerning, and I don't really see a way out. What affects me at this point though is certifications, and the same issue is kind of present there as well. I have a few colleagues who are AWS Certified xyz Engineers for example, but would stare at the AWS Management Console like a deer in the headlights, and would ask exceedingly stupid questions. The "fee extraction" practice wouldn't be too unfamiliar for the certification industry either - although that one doesn't bother me much, since I don't have to pay for these out of my own pocket, thankfully.

> If you learn the material, you pass and get a diploma. This is no more a gamble than your paycheck

I'd like to push back on this just a little bit. I'm sure it depends on where one lives, but here you either get your diploma or tough luck. There are no partial credentials. So while you can drop out (or just temporarily suspend your studies) at the end of semester, there's still stuff on the line. Not so much with a paycheck. I guess maybe a promotion is a closer analog, depending on how a given company does it (vibes vs something structured). This is further compounded by the social narrative, that if you don't get a degree then xyz, which is also not present for one's next monthly paycheck.

[0] What I guess I can mention is that I generally found the usual cycle of study season -> exam season to be very counter-productive. In general, all these "building up hype and then releasing it all at once" type situations were extremely taxing, and not for the right reasons. I think it's pretty agreeable at least that these do not result in good knowledge retention, do not inspire healthy student engagement, nor are actually necessary. Maybe this is not even a thing in better places, I don't know.

[1] I have absolutely no training in psychology or pedagogy, so take this with a mountain of salt, but I've found that people can be not just uninterested in learning, but grow downright hostile to it, often against their own self-recognized best interests. I've experienced it on myself, as well as seen it with others. It can be very difficult to snap someone out of such a state, and I have a lingering suspicion that it kind of forms a pipeline, with the lack of interest preceding it. I'm not sure that training and evaluating people in such a state results in a reasonable assessment, not for them, nor for the course they're taking.

Imagine otherwise: a teacher who wants change their final exam from a 50 item Scantron using A-D choices, to a 50 item Scantron using A-E choices, because they think having 5 choices per item is better than 4, would need to ask for IRB approval. That's not feasible, and is not what happens in the real world of academia.

It is true that local IRBs may try to add additional rules, but the NU policy you quote talks about "studies". Most IRBs would disagree that "professor playing around with grading procedures and policies" constitutes a "study".

It would be presumed exempted.

Are you a teacher or a student? If you are a teacher, you have wide latitude that a student researcher does not.

Also, if you are a teacher, doing "research about your teaching style", that's exempted.

By contrast, if you are a student, or a teacher "doing research" that's probably not exempt and must go through IRB.

The student is supposed to submit a whole conversation with an LLMs.

The LLM is prompted to answer a question or resolve a problem, and the LLM is there to assist. The LLM is instructed to never reveal the answer.

More interesting is the concept that the whole conversation is available to the instructor for grading. So if the LLMs makes mistake, or give away the solution, or if the student prompt engineer around it. It is all there and the instructor can take the necessary corrective measures.

87% of the students quite liked it, and we are looking forward to doubling the students that will be using it next quarter.

Overall, we are looking for more instructor to use it. So if you are interested in it please get in touch.

More info on: https://llteacher.blogspot.com/