Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fighting Fire with Fire: Scalable Oral Exams

behind-the-enemy-lines.com

Post reply on HN
Print view View original

151–160 of 288 posts

Re: Fighting Fire with Fire: Scalable Oral Exams

#151

Post by recursivecaveat »

NewsaHackO wrote: post #71
michaelt wrote: post #62

> We surveyed students before releasing grades to capture their experience. [...] Only 13% preferred the AI oral format. 57% wanted traditional written exams. [...] 83% of students found the oral exam framework more stressful than a written exam. [...] > Take-home exams are dead. Reverting to pen-and-paper exams in the classroom feels like a regression. Yeah, not sure the conclusion of the article really matches the…

The issue is that it is not scalable, unless there is some dependable, automated way to convert handwriting to text.

It is literally perfect linear scaling. For every student you must expend constant minutes of TA time grading the exam. Why is it unconscionable that the university should have an expense scale at the same rate it receives tuition revenue? $90,000 of tuition pays for a lot of grading hours. I feel that scalability is a cultural meme that has lost the plot.

Re: Fighting Fire with Fire: Scalable Oral Exams

#152

Post by sarchertech »

makeitdouble wrote: post #145
j_w wrote: post #127

Earlier quoted context omitted.

It's not that the oral format should be dismissed, just that the idea of your exam being speaking to a machine to be judged on the merit of your time in a course is dystopian. Talking to another human is fine.

How different is it in essence from checking boxes to be scanned by a machine and auto-evaluated to get a one dimention numerical score ? Have exams ever been about humanity and the optics of it ?

Very different. A scantron machine is deterministic and non-chaotic.

In addition to being non-deterministic LLMs can product vastly different output from very slightly different input.

That’s ignoring how vulnerable LLMs are to prompt injection, and if this becomes common enough that exams aren’t thoroughly vetted by humans, I expect prompt attacks to become common.

Also if this is about avoiding in person exams, what prevents students from just letting their AI talk to test AI.

Re: Fighting Fire with Fire: Scalable Oral Exams

#153

Post by geraldwhen »

jmye wrote: post #89
perching_aix wrote: post #77

Earlier quoted context omitted.

> Cohorts for programs with a thousand initial students had less than 10 graduates. This was the norm. And why is this a flex exactly? Almost sounds like fraud. Get sold on how you'll be taught well and become successful. Pay. Then be sent through an experience that filters so severely, only 1% of people pass. Receive 100% of the blame when you inevitably fail. Repeat for the other 990 students. The "university thank…

> And why is this a flex exactly? Almost sounds like fraud. Do you think you're just purchasing a diploma? Or do you think you're purchasing the opportunity to gain an education and potential certification that you received said education? It's entirely possible that the University stunk at teaching 99% of it's students (about as equally possible that 99% of the students stunk at learning), but "fraud" is absolute no…

In the modern era, you are purchasing a diploma. I witnessed dozens of students blatantly cheat without any consequence. We all got the same degree.

Colleges exist to collect tuition, especially from international students who pay more. Teaching anything at all, or punishing cheating, just isn’t that important.

Re: Fighting Fire with Fire: Scalable Oral Exams

#154

Post by baq »

jimbokun wrote: post #101
baq wrote: post #52

Earlier quoted context omitted.

> Plus this is more like how people work. if we want to educate people 'how people work', companies should be hiring interns and teaching them how people work. university education should be about education (duh) and deep diving into a few specialized topics, not job preparedness. AI makes this disconnect that much more obvious.

If that was the model all but a small handful of universities would be shut down tomorrow. It’s impossible to fund that many university degrees without the promise of increased earnings after completion.

So shut them down. What’s the point of having them anyway if the value proposition is only a long expensive internship with negative value outputs? Have the interns do actually useful stuff.

Re: Fighting Fire with Fire: Scalable Oral Exams

#155

Post by baq »

jimbokun wrote: post #114
viccis wrote: post #48

Earlier quoted context omitted.

Unless class sizes are astronomical, it's absurd to pay US tuition all to have a lazy professor who automates even the most human components of the education you're getting for that price. If the class cost me $50? Then sure, use Dr. Slop to examine my knowledge. But this professor's school charges them $90,000 a year and over $200k to get an MBA? Hell no!

Yes. At that point what’s the value add over using YouTube videos and ChatGPT on your own?

The certificate is the value as long as everyone trust it actually certifies what it says is certified. If a diploma can be had for promoting ChatGPT or Gemini a couple dozen times a year, trust in what it certifies should be rapidly eroding and universities should be scared because what you suggest is actually rational.

Re: Fighting Fire with Fire: Scalable Oral Exams

#156

Post by geraldwhen »

NewsaHackO wrote: post #79
acbart wrote: post #61

Earlier quoted context omitted.

I've had colleagues argue (prior to LLMs) that oral exams are superior to paper exams, for diagnosing understanding. I don't know how to validate that statement, but if the assumption is true than there is merit to finding a way to scale them. Not saying this is it, but I wouldn't say that it's fair to just dismiss oral exams entirely.

Yes, I hate oral exams, but they are definitely better at getting a whole picture of a person's understanding of topics. A lot of specialty boards in medicine do this. To me, the two issues are that it requires an experienced, knowledgeable, and empathetic examiner, who is able to probe the examinee about areas they seem to be struggling in, and paradoxically, its strength is in the fact that it is subjective. The ex…

Interviews have the same issues. But if you do anything more than read off templated questions like a robot, you can be accused of discrimination.

It is a sad world we live in.

Re: Fighting Fire with Fire: Scalable Oral Exams

#157

Post by TrackerFF »

lifetimerubyist wrote: post #60

This is all so crazy to me. I went to school long before LLMs were even a Google Engineer's brianfart for the transformer paper and the way I took exams was already AI proof. Everything hand written in pen in a proctored gymnasium. No open books. No computers or smart phones, especially ones connected to the internet. Just a department sanctioned calculator for math classes. I wrote assembly and C++ code by hand, and…

So did I, but a big difference today is the number of students, and how many of them are doing non-traditional programs. Lots and lots of online-only programs, offered through serious universities.

The old ways do not scale well once you pass a certain number of students.

Re: Fighting Fire with Fire: Scalable Oral Exams

#158

Post by jasonfarnon »

rfrey wrote: post #147
lifetimerubyist wrote: post #60

This is all so crazy to me. I went to school long before LLMs were even a Google Engineer's brianfart for the transformer paper and the way I took exams was already AI proof. Everything hand written in pen in a proctored gymnasium. No open books. No computers or smart phones, especially ones connected to the internet. Just a department sanctioned calculator for math classes. I wrote assembly and C++ code by hand, and…

I simply don't believe your university program had a 99% failure rate. Such a university should be shut down and sold for parts.

any private university, yes. I have seen state-supported universities in certain countries with very high failure rates for certain programs (I'm assuming 99% was an exaggeration for something more like "the vast majority failed").

Re: Fighting Fire with Fire: Scalable Oral Exams

#160

Post by jasonfarnon »

Traubenfuchs wrote: post #28
Arodex wrote: post #25

Earlier quoted context omitted.

Just a teleprompter is already enough to cheat at these, even filmed. With a two-way mirror correctly placed, you can look directly into the camera and look perfectly normal while reading. Next steps are bone conduction microphones, smart glasses, earrings... And the weeding out of anyone both honest and with social anxiety.

My cohort was actively working with invisible realy-inside ear speakers.

I have been wondering if some of my students who demonstrated zero knowledge in class but ace in-class exams were doing something like this. I figured something like a hacked out google glasses would do the trick.
Post reply on HN