Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fighting Fire with Fire: Scalable Oral Exams

behind-the-enemy-lines.com

Post reply on HN
Print view View original

121–130 of 288 posts

Re: Fighting Fire with Fire: Scalable Oral Exams

#121

Post by vasco »

jimbokun wrote: post #83
lifetimerubyist wrote: post #60

This is all so crazy to me. I went to school long before LLMs were even a Google Engineer's brianfart for the transformer paper and the way I took exams was already AI proof. Everything hand written in pen in a proctored gymnasium. No open books. No computers or smart phones, especially ones connected to the internet. Just a department sanctioned calculator for math classes. I wrote assembly and C++ code by hand, and…

Admitting 1000 students to get 10 graduates means there are morons in admissions doing zero vetting to make sure the students are qualified.

Or that there's morons teaching.

Re: Fighting Fire with Fire: Scalable Oral Exams

#122

Post by musicale »

vkou wrote: post #118
NewsaHackO wrote: post #71

Earlier quoted context omitted.

The issue is that it is not scalable, unless there is some dependable, automated way to convert handwriting to text.

I assure you, oral exams are completely scalable. But it does require most of a university's budget to go towards labs and faculty, and not administration and sports arenas and social services and vanity projects and three-star dorms.

One way of scaling out interactive/oral assessment (and personalized instruction in general) is to hire a group of course assistants/tutors from the previous cohort.

Re: Fighting Fire with Fire: Scalable Oral Exams

#123

Post by jimbokun »

semilin wrote: post #117
jimbokun wrote: post #110

Earlier quoted context omitted.

I would be annoyed that I can’t use AI to do my work but the instructor can have AI do his job.

Too bad. The premise should be that the instructor, by nature of having the position, already has understanding of the subject. As a student, you do not, and your goal is to gain it. Prompting an LLM to write a response for you does not build understanding. Therefore you should write unhindered by sophistry machines.

But the instructor is not applying their understanding in any way. By delegating the evaluation to AI, there is zero value add vs just asking ChatGPT to evaluate your knowledge and not paying $1000s or $10000s in tuition.

And universities wonder why enrollment is dropping.

Re: Fighting Fire with Fire: Scalable Oral Exams

#124

Post by musicale »

perching_aix wrote: post #77
lifetimerubyist wrote: post #60

This is all so crazy to me. I went to school long before LLMs were even a Google Engineer's brianfart for the transformer paper and the way I took exams was already AI proof. Everything hand written in pen in a proctored gymnasium. No open books. No computers or smart phones, especially ones connected to the internet. Just a department sanctioned calculator for math classes. I wrote assembly and C++ code by hand, and…

> Cohorts for programs with a thousand initial students had less than 10 graduates. This was the norm. And why is this a flex exactly? Almost sounds like fraud. Get sold on how you'll be taught well and become successful. Pay. Then be sent through an experience that filters so severely, only 1% of people pass. Receive 100% of the blame when you inevitably fail. Repeat for the other 990 students. The "university thank…

If teaching is the goal, a 99% failure rate seems counterproductive.

Re: Fighting Fire with Fire: Scalable Oral Exams

#125

Post by themantalope »

beezlebroxxxxxx wrote: post #51
dvh wrote: post #27

Students cheat when grades are more valuable than knowledge.

This is not hitting the problem. Most students in universities are completely fine with awful grades or expect comical levels of grade inflation. Ask a professor or TA and you'll hear about an insane level of entitlement from students after they hand in extremely shoddy work. Failing students is actually quite hard or extremely discouraged by admins. The real problem is students and universities have collectively bou…

There is a lot in this comment I agree with, however I think may universities have backed themselves into a corner with the degree of tuition inflation that has taken place over the last 20+ years.

I graduated from a SUNY school in 2012. At the time, you could still actually go to school and work part time and get through it. Not saying it was easy by any stretch but it was possible. Tuition + living expenses were about $17/year on campus , less expensive housing was available off campus.

Now, even state schools have tuition which is only affordable through family wealth or loans. Going to university is no longer a low stakes choice - if you flunk you’re stuck with that debt forever. Not to say students aren’t responsible for understanding that when signing up, but the stakes are just a lot higher than what it used to be.

Re: Fighting Fire with Fire: Scalable Oral Exams

#126

Post by knallfrosch »

BoiledCabbage wrote: post #68
michaelt wrote: post #62

> We surveyed students before releasing grades to capture their experience. [...] Only 13% preferred the AI oral format. 57% wanted traditional written exams. [...] 83% of students found the oral exam framework more stressful than a written exam. [...] > Take-home exams are dead. Reverting to pen-and-paper exams in the classroom feels like a regression. Yeah, not sure the conclusion of the article really matches the…

That's what so surprising to me - they data clearly shows the experiment had terrible results. And the write up is nothing but the author stating: "glowing success!". And they didn't even bother to test the most important thing. Were the LLM evaluations even accurate! Have graders manually evaluate them and see if the LLMs were even close or were wildly off. This is clearly someone who had a conclusion to promote reg…

I found "well, the LLMs converge when given each other's scores, so they agree and are correct" to be quite the jump to a conclusion.

Re: Fighting Fire with Fire: Scalable Oral Exams

#127

Post by j_w »

vasco wrote: post #120
michaelt wrote: post #62

> We surveyed students before releasing grades to capture their experience. [...] Only 13% preferred the AI oral format. 57% wanted traditional written exams. [...] 83% of students found the oral exam framework more stressful than a written exam. [...] > Take-home exams are dead. Reverting to pen-and-paper exams in the classroom feels like a regression. Yeah, not sure the conclusion of the article really matches the…

One student had to talk to an AI for more than 60 minutes. These guys are creating a dystopia. Also students will just have an AI pick up the phone if this gets used for more than 2 semesters.

It's not that the oral format should be dismissed, just that the idea of your exam being speaking to a machine to be judged on the merit of your time in a course is dystopian. Talking to another human is fine.

Re: Fighting Fire with Fire: Scalable Oral Exams

#128

Post by [deleted] »

perching_aix wrote: post #77
lifetimerubyist wrote: post #60

This is all so crazy to me. I went to school long before LLMs were even a Google Engineer's brianfart for the transformer paper and the way I took exams was already AI proof. Everything hand written in pen in a proctored gymnasium. No open books. No computers or smart phones, especially ones connected to the internet. Just a department sanctioned calculator for math classes. I wrote assembly and C++ code by hand, and…

> Cohorts for programs with a thousand initial students had less than 10 graduates. This was the norm. And why is this a flex exactly? Almost sounds like fraud. Get sold on how you'll be taught well and become successful. Pay. Then be sent through an experience that filters so severely, only 1% of people pass. Receive 100% of the blame when you inevitably fail. Repeat for the other 990 students. The "university thank…

[deleted]

Re: Fighting Fire with Fire: Scalable Oral Exams

#129

Post by j_w »

semilin wrote: post #111
dvh wrote: post #27

Students cheat when grades are more valuable than knowledge.

I think this points to the only real sustainable solution: make it so that students would prefer to do real work. We have seen for ages the distinction between seeming and being in regards to verbal understanding blurred. LLMs are only an acceleration of the blurring. Therefore it will at some point become essentially impossible to determine whether one really understands something. The two solutions to this are (1)…

Or you just make in person exams the majority of the work and make the exams brutal. If you can't pass the exams you don't pass the class, so you need to learn enough to pass the exams.

Re: Fighting Fire with Fire: Scalable Oral Exams

#130

Post by Nifty3929 »

michaelt wrote: post #88
cvoss wrote: post #82

Earlier quoted context omitted.

The quote you gave is not the conclusion of the article. It's a self-evident claim that just as well could have been the first sentence of the article ("take-home exams are dead"), followed by an opinion ("reverting ... feels like a regression") which motivated the experiment. Some universities and professors have tried to move to a take-home exam format, which allows for more comprehensive evaluation with easier log…

> The quote you gave is not the conclusion of the article. Did I say "conclusion" ? Sorry, I should have said the section just before the acknowledgements, where the conclusion would normally be, entitled "The bigger point"

I think this is the actual conclusion: "Now, AI is making them scalable again."

That is, the author concluded that AI tools provide viable alternatives to the other available options, and which solve many of their problems.

Post reply on HN