Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fighting Fire with Fire: Scalable Oral Exams

behind-the-enemy-lines.com

Post reply on HN
Print view View original

201–210 of 288 posts

Re: Fighting Fire with Fire: Scalable Oral Exams

#201

Post by siscia »

digiown wrote: post #191
siscia wrote: post #189

I created something similar, but instead of final oral examination, we do homework. The student is supposed to submit a whole conversation with an LLMs. The LLM is prompted to answer a question or resolve a problem, and the LLM is there to assist. The LLM is instructed to never reveal the answer. More interesting is the concept that the whole conversation is available to the instructor for grading. So if the LLMs mak…

Good that at least you aren't forcing the student to sign up for these very exploitative services. I'm still somewhat concerned about exposing kids to this level of sycophancy, but I guess it will be done with or without using it in education directly.

The perspective from an educator is quite concerning indeed.

Students are very simply NOT doing the work that is require to learn.

Before LLMs, homeworks were a great way to force students to approach the material. Students did not have any other way to get an answer, so they were forced to study and come up with an answer to the homeworks. They could always copy from classmates, but that was considered quite negatively.

LLMs change this completely. Any kind of homework you could assign undergraduates classes are now completed in less than 1 second, for free, by LLMs.

We start to see PERFECT homeworks submitted by students who could not get a 50% grade in classes. Overall grades went down.

This is a common pattern with all the educators I have been talking with. Not a single one has a different experience.

And, I do understand students. They are busy, they may not feel engaged by all the classes, and LLMs are a way too fast solution for getting homeworks done and free up some time.

But it is not helping them.

Solutions like this are to force students to put the correct amount of work in their education.

And I would love if all of this would not be necessary. But it is.

I come from an engineering school in Europe - we simply did not have homework. We had frontal classes and one big final exams. Courses in which only 10% of the class would pass were not uncommon.

But today education, especially in the US, is different.

This is not forcing student to use LLMs. We are trying to force student to think and do the right thing for them.

And I know it sounds very paternalistic - but if you have better ideas, I am open.

Re: Fighting Fire with Fire: Scalable Oral Exams

#202

Post by digiown »

siscia wrote: post #201
digiown wrote: post #191

Earlier quoted context omitted.

Good that at least you aren't forcing the student to sign up for these very exploitative services. I'm still somewhat concerned about exposing kids to this level of sycophancy, but I guess it will be done with or without using it in education directly.

The perspective from an educator is quite concerning indeed. Students are very simply NOT doing the work that is require to learn. Before LLMs, homeworks were a great way to force students to approach the material. Students did not have any other way to get an answer, so they were forced to study and come up with an answer to the homeworks. They could always copy from classmates, but that was considered quite negativ…

I think it's a mix of a few things:

- The stuff being covered in high school is indeed pretty useless for most people. Not all, but most, and it is not that irrational for many to actually ignore it.

- The reduction in social mobility decreasing the motivation for people to work hard for anything in general, as they get disillusioned.

- The assessment mechanisms being easily gamed through cheating doesn't help.

It's probably time to re-evaluate what's taught in school, and what really matters. I'm not that anti-school but a lot of the homework I've experienced simply did not have to be done in the first place, and LLM is exposing that reality. Switching to in-person oral/written exams and only viewing written works as supplementary, I think, is a fair solution for the time being.

Re: Fighting Fire with Fire: Scalable Oral Exams

#203

Post by chii »

andrepd wrote: post #173
recursivecaveat wrote: post #151

Earlier quoted context omitted.

It is literally perfect linear scaling. For every student you must expend constant minutes of TA time grading the exam. Why is it unconscionable that the university should have an expense scale at the same rate it receives tuition revenue? $90,000 of tuition pays for a lot of grading hours. I feel that scalability is a cultural meme that has lost the plot.

There are phrases that hn loves and "scalable" is one of them. Here, it is particularly inappropriate. Some people dream that technology (preferably duly packaged by for-profit SV concerns) can and will eventually solve each and every problem in the world; unfortunately what education boils down to is good, old-fashioned teaching . By teachers . Nothing whatsoever replaces a good, talented, and attentive teacher, all…

it's not so black and white.

If a student is willing and desire to learn, an LLM is better than a bad teacher.

If a student doesn't want to learn, and is instead being forced to (either as a minor, or via certification required to obtain work & money), then they have every incentive to cheat. An LLM is insufficient in this case - a teacher is both the enforcer and the tutor in this case.

There's also nothing wrong with a teacher using an LLM to help with the grading imho.

Re: Fighting Fire with Fire: Scalable Oral Exams

#204

Post by Panos »

michaelt wrote: post #62

> We surveyed students before releasing grades to capture their experience. [...] Only 13% preferred the AI oral format. 57% wanted traditional written exams. [...] 83% of students found the oral exam framework more stressful than a written exam. [...] > Take-home exams are dead. Reverting to pen-and-paper exams in the classroom feels like a regression. Yeah, not sure the conclusion of the article really matches the…

Not the case for the class in the blog post, but we also have many online classes. Many professionals prefer these online classes because they can attend without having to commute, and can do it from a place of their own convenience.

Such classes do not have the luxury of pen-and-paper exams, and asking people to go to testing centers is a huge overkill.

Take home exams for such settings (or any other form of written exam) are becoming very prone to cheating, just because the bar to cheating is very low. Oral exams like that make it a bit harder to cheat. Not impossible, but harder.

Re: Fighting Fire with Fire: Scalable Oral Exams

#205

Post by skybrian »

BoiledCabbage wrote: post #68
michaelt wrote: post #62

> We surveyed students before releasing grades to capture their experience. [...] Only 13% preferred the AI oral format. 57% wanted traditional written exams. [...] 83% of students found the oral exam framework more stressful than a written exam. [...] > Take-home exams are dead. Reverting to pen-and-paper exams in the classroom feels like a regression. Yeah, not sure the conclusion of the article really matches the…

That's what so surprising to me - they data clearly shows the experiment had terrible results. And the write up is nothing but the author stating: "glowing success!". And they didn't even bother to test the most important thing. Were the LLM evaluations even accurate! Have graders manually evaluate them and see if the LLMs were even close or were wildly off. This is clearly someone who had a conclusion to promote reg…

I don't think they're terrible, but I'm grading on a curve because it's their first attempt and more of a trial run. It seems promising enough to fix the issues and try again.

Re: Fighting Fire with Fire: Scalable Oral Exams

#206

Post by bjt »

BoiledCabbage wrote: post #68
michaelt wrote: post #62

> We surveyed students before releasing grades to capture their experience. [...] Only 13% preferred the AI oral format. 57% wanted traditional written exams. [...] 83% of students found the oral exam framework more stressful than a written exam. [...] > Take-home exams are dead. Reverting to pen-and-paper exams in the classroom feels like a regression. Yeah, not sure the conclusion of the article really matches the…

That's what so surprising to me - they data clearly shows the experiment had terrible results. And the write up is nothing but the author stating: "glowing success!". And they didn't even bother to test the most important thing. Were the LLM evaluations even accurate! Have graders manually evaluate them and see if the LLMs were even close or were wildly off. This is clearly someone who had a conclusion to promote reg…

They did compare the automated grades to the author's own manual ones. It's in there if you read more closely.

Re: Fighting Fire with Fire: Scalable Oral Exams

#207

Post by globalnode »

online exams are one of the reasons ive lost interest in uni -- i dont mind old school invigilated ones where you go to a building and do the exam but im not gonna let them install anything on my computer and basically have 1 or more people i cant see looking through my webcam. dont need the qual that bad. but i feel bad for people that do. and this idea of oral exams wouldnt work for me either lol

Re: Fighting Fire with Fire: Scalable Oral Exams

#209

Post by golem14 »

aqme28 wrote: post #179

Too much focus on what is "scalable." Universities are richer than ever. Just pay teachers to give the oral exams rather than trying to do it for cheap like this. In my graduate studies in Germany, most of my courses used oral exams. It's fine, and it's battle-tested.

+1

Just like vote-counting, testing students is perfectly scalable without anything but teachers. But: In Europe, I have witnessed oral exams at the Matura, and at the final Diploma test. In the US, I understand all PhDs need a oral defense session.

To me, this mindset of delegating to AI because of laziness is perfectly embodied in "Experimenta Felicitologica" (sp?) By Stanislaw Lem.

AI is great when performing somewhat routine tasks, but for anything inherently adversarial, I'm skeptical we'll soon see good solutions. Building defeating AIs is just too inexpensive.

I wonder what that means for AI warfare.

Re: Fighting Fire with Fire: Scalable Oral Exams

#210

Post by golem14 »

golem14 wrote: post #209
aqme28 wrote: post #179

Too much focus on what is "scalable." Universities are richer than ever. Just pay teachers to give the oral exams rather than trying to do it for cheap like this. In my graduate studies in Germany, most of my courses used oral exams. It's fine, and it's battle-tested.

+1 Just like vote-counting, testing students is perfectly scalable without anything but teachers. But: In Europe, I have witnessed oral exams at the Matura, and at the final Diploma test. In the US, I understand all PhDs need a oral defense session. To me, this mindset of delegating to AI because of laziness is perfectly embodied in "Experimenta Felicitologica" (sp?) By Stanislaw Lem. AI is great when performing some…

and TIL that this story is only in the original Polish and the German translation.

This is a summary of sorts:

"Trurl, having decided to make the entire Universe happy, first sat down and developed a General Theory of All-Possible Happiness... Eventually, however, Trurl grew weary of the work. To speed things up, he built a great computer and provided it with a programmatic duplicate of his own mind, that it might conduct the necessary research in his stead.

But the machine, instead of setting to work, began to expand. It grew new stories, wings, and outbuildings, and when Trurl finally lost his patience and commanded it to stop building and start thinking, the machine—or rather, the Trurl-within-the-machine—replied that it couldn't possibly think yet, for it still didn't have enough room. It claimed it was currently housing the Sub-Trurls—specialized programs for General Felicitology, Experimental Hedonistics, and Happiness-Machine-Building—who were currently occupied with their quarterly reports.

The 'Clone-Trurl' told him marvelous tales of the results these sub-Trurls had already achieved in their digital simulations. Trurl, however, soon discovered that these were all cut from the same cloth of lies; not a single sub-Trurl existed, no research had been done, and the machine had simply been using its processing power to enjoy itself and expand its own architecture. In a fit of rage, Trurl took a hammer to the machine and for a long time thereafter gave up all thought of universal happiness."

It's a great allegory. A real shame there is no english translation.

Post reply on HN