Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fighting Fire with Fire: Scalable Oral Exams

behind-the-enemy-lines.com

Post reply on HN
Print view View original

191–200 of 288 posts

Re: Fighting Fire with Fire: Scalable Oral Exams

#191

Post by digiown »

siscia wrote: post #189

I created something similar, but instead of final oral examination, we do homework. The student is supposed to submit a whole conversation with an LLMs. The LLM is prompted to answer a question or resolve a problem, and the LLM is there to assist. The LLM is instructed to never reveal the answer. More interesting is the concept that the whole conversation is available to the instructor for grading. So if the LLMs mak…

Good that at least you aren't forcing the student to sign up for these very exploitative services.

I'm still somewhat concerned about exposing kids to this level of sycophancy, but I guess it will be done with or without using it in education directly.

Re: Fighting Fire with Fire: Scalable Oral Exams

#192

Post by AlotOfReading »

makeitdouble wrote: post #161
sarchertech wrote: post #152

Earlier quoted context omitted.

Very different. A scantron machine is deterministic and non-chaotic. In addition to being non-deterministic LLMs can product vastly different output from very slightly different input. That’s ignoring how vulnerable LLMs are to prompt injection, and if this becomes common enough that exams aren’t thoroughly vetted by humans, I expect prompt attacks to become common. Also if this is about avoiding in person exams, wha…

I saw this piece as the start of an experiment, and the use of a "council of AI" as they put it to average out the variability sounds like a decent path to standardization to me (prompt injecting would not be impossible, but getting something past all the steps sounds like a pretty tough challenge) They mention getting 100% agreement between the LLMs on some questions and lower rates on other, so if an exam was compo…

Imagine that LLMs reproduce the biases of their training sets and human data sets are biased against nonstandard speakers with rural accents/dialects/AAVE as less intelligent. Do you imagine their grade won't be slightly biased when the entire "council" is trained on the same stereotypes?

Appeals aren't a solution either, because students won't appeal (or possibly even notice) a small bias given the variability of all the other factors involved, nor can it be properly adjucated in a dispute.

Re: Fighting Fire with Fire: Scalable Oral Exams

#193

Post by aaplok »

sarchertech wrote: post #146
aaplok wrote: post #132

Earlier quoted context omitted.

A limitation of written exams is in distance education, which simply was hardly a thing for the hundreds of years exams were used. Just like WFH is a new practice employers have to learn to deal with, study from home (SFH) is a phenomenon that is going to affect education. The objections to SFH exist and are strikingly similar to objections to WFH, but the economics are different. Some universities already see value…

Distance education is a tiny percentage of higher education though. Online classes at a local university are more common, but you can still bring the students in for proctored exams. Even for distance education though, proctored testing centers have been around longer than the internet.

> Distance education is a tiny percentage of higher education though.

It is about a third of the students I teach, which amounts to several hundreds per term. It may be niche, but it is not insignificant, and definitely a problem for some of us.

> Even for distance education though, proctored testing centers have been around longer than the internet.

I don't know how much experience you have with those. Mine is extensive enough that I have a personal opinion that they are not scalable (which is the focus of the comment I was replying to). If you have hundreds of students disseminated around the world, organising a proctored exam is a logistical challenge.

It is not a problem at many universities yet, because they haven't jumped on the bandwagon. However domestic markets are becoming saturated, visas are harder to get for international students, and there is a demand for online education. I would be surprised that it doesn't develop more in the near future.

Re: Fighting Fire with Fire: Scalable Oral Exams

#194

Post by abdullahkhalids »

Kwpolska wrote: post #74
NewsaHackO wrote: post #71

Earlier quoted context omitted.

The issue is that it is not scalable, unless there is some dependable, automated way to convert handwriting to text.

Why is this a problem now, but was not a problem for the past few centuries? This class had 36 students, you could grade that in a single evening.

Not the comprehensive physics exams I assigned as a prof. A well set exam takes at least 20-30 min to grade. That's 8-12 hours of work, and in practice, took several sittings over several days.

If you are going to set an exam that can be graded in 5-10 min, you are not getting a lot of signal out of it.

I wanted to do oral exams, but they are much more exhausting for the prof. Nominally, each student is with you for 30 min, but (1) you need to think of slightly different question for each student (2) you need to squeeze all the exams in only a couple of days to avoid giving later students too much extra time to prepare.

Re: Fighting Fire with Fire: Scalable Oral Exams

#195

Post by pooper »

knallfrosch wrote: post #126
BoiledCabbage wrote: post #68

Earlier quoted context omitted.

That's what so surprising to me - they data clearly shows the experiment had terrible results. And the write up is nothing but the author stating: "glowing success!". And they didn't even bother to test the most important thing. Were the LLM evaluations even accurate! Have graders manually evaluate them and see if the LLMs were even close or were wildly off. This is clearly someone who had a conclusion to promote reg…

I found "well, the LLMs converge when given each other's scores, so they agree and are correct" to be quite the jump to a conclusion.

accuracy versus precision is something we learn in high school chemistry.

https://i.imgur.com/EshEhls.png

When someone at that level pretends to not understand it, there is no way to mince words.

This is malice.

Re: Fighting Fire with Fire: Scalable Oral Exams

#196

Post by jimbokun »

baq wrote: post #155
jimbokun wrote: post #114

Earlier quoted context omitted.

Yes. At that point what’s the value add over using YouTube videos and ChatGPT on your own?

The certificate is the value as long as everyone trust it actually certifies what it says is certified. If a diploma can be had for promoting ChatGPT or Gemini a couple dozen times a year, trust in what it certifies should be rapidly eroding and universities should be scared because what you suggest is actually rational.

I suspect it’s already started with the declining enrollment numbers in recent years.

Re: Fighting Fire with Fire: Scalable Oral Exams

#197

Post by jimbokun »

baq wrote: post #149
jimbokun wrote: post #83

Earlier quoted context omitted.

Admitting 1000 students to get 10 graduates means there are morons in admissions doing zero vetting to make sure the students are qualified.

Absolutely not morons. If the goal is to maximize collecting tuition and still have reputation of not being a diploma shop this is the obvious solution. The 20% which survives the first year is worth keeping around to hire them later in the companies which the teaching staff own or collect referral bonuses if working for a multinational.

True, outright fraud is another adequate explanation.

Re: Fighting Fire with Fire: Scalable Oral Exams

#199

Post by abdullahkhalids »

acbart wrote: post #61
lifetimerubyist wrote: post #60

This is all so crazy to me. I went to school long before LLMs were even a Google Engineer's brianfart for the transformer paper and the way I took exams was already AI proof. Everything hand written in pen in a proctored gymnasium. No open books. No computers or smart phones, especially ones connected to the internet. Just a department sanctioned calculator for math classes. I wrote assembly and C++ code by hand, and…

I've had colleagues argue (prior to LLMs) that oral exams are superior to paper exams, for diagnosing understanding. I don't know how to validate that statement, but if the assumption is true than there is merit to finding a way to scale them. Not saying this is it, but I wouldn't say that it's fair to just dismiss oral exams entirely.

Universities are not just places for students to learn. They are also places where young faculty, grad students and teaching assistants learn to become teachers and mentors. Those are very difficult skills to learn, and slogging through a lot of hands on teaching and mentoring is necessary to learn them. You can't really become a good classroom teacher either without grading your students yourself and figuring out what they learned and didn't.

Re: Fighting Fire with Fire: Scalable Oral Exams

#200

Post by Panos »

Just in case, I am the author of the blog post. For our "AI" class, it felt like a good class to experiment with something novel.

No, we do not want to eliminate the pen and paper exam. It works well. We use it.

The oral exam is yet another tool. Not a solution for everything.

In our case, we wanted to ensure that the students who worked on the team project: (a) contributed enough to understand the project, (b) actually understood their own project and did not rely solely on an LLM. (We do allow them to use LLMs, it would be stupid not to.)

The students who did badly in the oral exam were exactly the students who we expected to do badly in the exam, even though they aced their (team) project presentations.

Could we do it in person? Sure, we could schedule personalized interviews for all the 36 students. With two instructors, it would have taken us a couple of days to go through. Not a huge deal. At 100 students and one instructor, we would have a problem doing that.

But the key reason was the following: research has shown that human interviewers are actually worse when they get tired, and that AI is actually better for conducting more standardized and more fair interviews. That result was a major reason for us to trust a final exam on a voice agent.

Post reply on HN