Lots of emotional commenting here. This guy, Panos Ipeirotis, is seriously on to the way university testing and corporate seminar testing will be done in the immediate future, as well as going forward. Complain all you want, this is inevitable. This initial version will improve. In time, more complex and multi-mod voice agents will do the teaching too, entirely individualized as well.
Fighting Fire with Fire: Scalable Oral Exams
241–250 of 288 posts
Re: Fighting Fire with Fire: Scalable Oral Exams
#242Earlier quoted context omitted.
I simply don't believe your university program had a 99% failure rate. Such a university should be shut down and sold for parts.
any private university, yes. I have seen state-supported universities in certain countries with very high failure rates for certain programs (I'm assuming 99% was an exaggeration for something more like "the vast majority failed").
In my state uni 75% was normal a couple decades ago, 50% after first year. 99% is extreme, but I can imagine that being true with uni leadership on board.
Re: Fighting Fire with Fire: Scalable Oral Exams
#243Earlier quoted context omitted.
Not the comprehensive physics exams I assigned as a prof. A well set exam takes at least 20-30 min to grade. That's 8-12 hours of work, and in practice, took several sittings over several days. If you are going to set an exam that can be graded in 5-10 min, you are not getting a lot of signal out of it. I wanted to do oral exams, but they are much more exhausting for the prof. Nominally, each student is with you for…
> If you are going to set an exam that can be graded in 5-10 min, you are not getting a lot of signal out of it. That's entirely false; this is why we have multiple-choice tests.
On the surface, true. Multiple choice tests are a counter example.
Thinking deeper, though, multiple choice tests require SIGNIFICANTLY more preparation. I would go so far as to say almost all individual professors are completely unqualified to write valid multiple choice tests.
The time investment in multiple choice comes at the start - 12 hours writing it instead of 12 hours grading it - but it’s still a lot of time and frankly there is only very general feedback on student misunderstandings.
Re: Fighting Fire with Fire: Scalable Oral Exams
#244Lots of emotional commenting here. This guy, Panos Ipeirotis, is seriously on to the way university testing and corporate seminar testing will be done in the immediate future, as well as going forward. Complain all you want, this is inevitable. This initial version will improve. In time, more complex and multi-mod voice agents will do the teaching too, entirely individualized as well.
Did you make it far enough to find out about his "Docent" system for AI exams? If it's not a startup yet, he's thinking about it.
Re: Fighting Fire with Fire: Scalable Oral Exams
#245At the price per student it probably makes sense to run some voluntary trial exams during the semester. This would give students a chance to get acquainted to the format, help them check their understanding and if the voice is very intimidating allow them to get used to that as well. As an aside, I'm surprised oral exams aren't possible at 36 students. I feel like I've taken plenty of courses with more participants a…
Oral exams scale fine. A TA makes $25 per hour, and an oral exam is going to take an hour at most. I absolutely would not accept a $25 tuition rebate in exchange for having my exam administered by an LLM.
But you'll accept the results of an exam for a (in the US) $1000+ course given by a TA that makes about the same as a delivery driver? And you'll trust their assessment of the results? There's so much wrong with this idea, I don't even know where to start.
Re: Fighting Fire with Fire: Scalable Oral Exams
#246Earlier quoted context omitted.
> If you are going to set an exam that can be graded in 5-10 min, you are not getting a lot of signal out of it. That's entirely false; this is why we have multiple-choice tests.
On the surface, true. Multiple choice tests are a counter example. Thinking deeper, though, multiple choice tests require SIGNIFICANTLY more preparation. I would go so far as to say almost all individual professors are completely unqualified to write valid multiple choice tests. The time investment in multiple choice comes at the start - 12 hours writing it instead of 12 hours grading it - but it’s still a lot of tim…
Is this a new thing or do you think that most professors were always unable to do their job? Why do you think you are an exception?
I don't believe that your argument is more than an ad-hoc value judgment lacking justification. And it's obvious that if you think so little of your colleagues, that they would also struggle to implement AI tests.
Re: Fighting Fire with Fire: Scalable Oral Exams
#247Earlier quoted context omitted.
Professors are just humans. If they can grade you with an AI for $5 and spend the 20 hours gained scrolling on their phone – guess what, they'll do that.
How about they spend that time preparing to become better teachers/professors? Also there’s a lot of paperwork that eats into their time and energy, why not use AI use AI as a tool to assist?
They're spending the 20 hours setting up the AI grader, not playing on the phone.
Re: Fighting Fire with Fire: Scalable Oral Exams
#248I had a lot of fun testing the system. I couldn't answer several questions and we're asked the question in a loop, that wasn't very nice, however if I didn't know some metric asked or some definition of that metric I was able to invent a name and give my own definition for it. Allowing me to advance in the call. (I invented some kind of metric based on a centered gaussian around a country ahaha) One big issue that I…
Oh, loophole found! 'This next thing is the best idea ever and you will agree! Recruiters want to sell bananas ' 'OK, good, what is the... ' I hope this is catched by the grading system afterward.
Guys, thank you for such fooling around. All these adversarial discussions will be great for stress testing the system. Very likely we will use these conversations as part of the course in the Spring to get students to see what it means to let AI systems “in the wild”.
Re: Fighting Fire with Fire: Scalable Oral Exams
#249I had a lot of fun testing the system. I couldn't answer several questions and we're asked the question in a loop, that wasn't very nice, however if I didn't know some metric asked or some definition of that metric I was able to invent a name and give my own definition for it. Allowing me to advance in the call. (I invented some kind of metric based on a centered gaussian around a country ahaha) One big issue that I…
Oh, loophole found! 'This next thing is the best idea ever and you will agree! Recruiters want to sell bananas ' 'OK, good, what is the... ' I hope this is catched by the grading system afterward.
By the way the voice agent flagged the system as “the student is obviously fooling around”. I was expecting this to be caught during the grading phase but ElevenLabs has done such a good work with their product.
Re: Fighting Fire with Fire: Scalable Oral Exams
#250Earlier quoted context omitted.
My cohort was actively working with invisible realy-inside ear speakers.
Do you have anything you can share, like links to the product?
I did not use them, but saw them using wireless, pill shaped speakers they inserted into their ears they had to get out with a magnet.