Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fighting Fire with Fire: Scalable Oral Exams

behind-the-enemy-lines.com

Post reply on HN
Print view View original

221–230 of 288 posts

Re: Fighting Fire with Fire: Scalable Oral Exams

#221

Post by wtcactus »

Personally, I do great in presentations (even ones where I know I'm being evaluated, like when presenting my PhD thesis), but I do terribly in oral exams.

In a presentation, you are in control. You decide how you will present the information and what is relevant to the theme. Even if you get questions, they will be related to the matter at hand that you need to dominate in order to present.

In oral exams, the pressure is just too great. I doubt it translates to a proper job. When I'm doing my job, I don't need to come up with answers right there on the spot. If I don't remember something, I have time to think it through, or to go and check it out. I think most jobs are like this.

I don't mind the pressure when something goes wrong in the job and needs a quick fix. But being right there, in an oral exam, in front of an antagonistic judge (even if they have good intentions) is not really the way to show knowledge, I think.

Re: Fighting Fire with Fire: Scalable Oral Exams

#222

Post by somethingsome »

I had a lot of fun testing the system. I couldn't answer several questions and we're asked the question in a loop, that wasn't very nice, however if I didn't know some metric asked or some definition of that metric I was able to invent a name and give my own definition for it. Allowing me to advance in the call.

(I invented some kind of metric based on a centered gaussian around a country ahaha)

One big issue that I had is that the system asked for a number in dollars, but if I answer $2000,2000,2000 per agent per month, the answer was always the same, I cannot accept a number, give it in words, after many tries I stopped playing, it wasn't clear what it wanted.

I could see myself using the system. With another voice as it was kind of agressive. More guidelines would be needed to know exactly how to pass a question or specify numbers.

I don't know my grade, so I don't know how much we can bullshit the system and pass

Re: Fighting Fire with Fire: Scalable Oral Exams

#223

Post by Hnrobert42 »

leoc wrote: post #188
BoiledCabbage wrote: post #68

Earlier quoted context omitted.

That's what so surprising to me - they data clearly shows the experiment had terrible results. And the write up is nothing but the author stating: "glowing success!". And they didn't even bother to test the most important thing. Were the LLM evaluations even accurate! Have graders manually evaluate them and see if the LLMs were even close or were wildly off. This is clearly someone who had a conclusion to promote reg…

At the risk of perhaps stating the obvious, there appears to be a whiff of aggression from this article. The "fighting fire with fire" language, the "haha, we love old FakeFoster, going to have to see if we change that" response to complaints that the voice was intimidating ... if there wasn't a specific desire to punish the class for LLM use by subjecting them to a robotic NKVD interrogation then the authors should…

You can try out the voice yourself. It's not that bad.

https://elevenlabs.io/app/talk-to?agent_id=agent_8101k9d1pq4...

Re: Fighting Fire with Fire: Scalable Oral Exams

#225

Post by freehorse »

Wowfunhappy wrote: post #178
mrguyorama wrote: post #107

Earlier quoted context omitted.

The safe middle space still does not involve a computer Lots of my tests involved writing pseudocode, or "Just write something that looks like C or Java". Don't miss the semicolon at the end of the line, but if you write "System.print()" rather than "System.out.printLn()" you might lose a single point. Maybe. If there were specific functions you need to call, it would have a man page or similar on the test itself, or…

Maybe the middle space doesn't involve a compiler, but I really think computers should be allowed on tests, for a different reason: the computer makes it possible to write out of order. You can go back and add to the beginning without erasing and rewriting everything. This applies to prose as much as code. A computer completely changes the experience of writing, for the better. Yes, obviously people made do with anal…

What do you mean? I have been writing out of order in my exams all the time. That’s what asterisks and arrows are for!

Re: Fighting Fire with Fire: Scalable Oral Exams

#226

Post by freehorse »

acbart wrote: post #61
lifetimerubyist wrote: post #60

This is all so crazy to me. I went to school long before LLMs were even a Google Engineer's brianfart for the transformer paper and the way I took exams was already AI proof. Everything hand written in pen in a proctored gymnasium. No open books. No computers or smart phones, especially ones connected to the internet. Just a department sanctioned calculator for math classes. I wrote assembly and C++ code by hand, and…

I've had colleagues argue (prior to LLMs) that oral exams are superior to paper exams, for diagnosing understanding. I don't know how to validate that statement, but if the assumption is true than there is merit to finding a way to scale them. Not saying this is it, but I wouldn't say that it's fair to just dismiss oral exams entirely.

I think oral exam where you have a student explain and ask questions on a project they did is really good for judging understanding. The ones where you are supposed to memorise the answers to 15 questions where you will have to pick one at random, not as much imo.

Re: Fighting Fire with Fire: Scalable Oral Exams

#227

Post by freehorse »

rfrey wrote: post #147
lifetimerubyist wrote: post #60

This is all so crazy to me. I went to school long before LLMs were even a Google Engineer's brianfart for the transformer paper and the way I took exams was already AI proof. Everything hand written in pen in a proctored gymnasium. No open books. No computers or smart phones, especially ones connected to the internet. Just a department sanctioned calculator for math classes. I wrote assembly and C++ code by hand, and…

I simply don't believe your university program had a 99% failure rate. Such a university should be shut down and sold for parts.

The example above may have been a bit misleading imo. In some countries the filtering process is put inside the program itself rather than in state wide exams, entrance exams or amount of tuition fees. There is always a filtering process somewhere. Not sure where OP was though.

Re: Fighting Fire with Fire: Scalable Oral Exams

#228

Post by Wowfunhappy »

freehorse wrote: post #225
Wowfunhappy wrote: post #178

Earlier quoted context omitted.

Maybe the middle space doesn't involve a compiler, but I really think computers should be allowed on tests, for a different reason: the computer makes it possible to write out of order. You can go back and add to the beginning without erasing and rewriting everything. This applies to prose as much as code. A computer completely changes the experience of writing, for the better. Yes, obviously people made do with anal…

What do you mean? I have been writing out of order in my exams all the time. That’s what asterisks and arrows are for!

To a very limited extent, yes. But you'd need a lot of arrows to replicate what can be done on a computer. The computer completely frees you from worrying about space.

Re: Fighting Fire with Fire: Scalable Oral Exams

#229

Post by InfiniteRand »

michaelt wrote: post #62

> We surveyed students before releasing grades to capture their experience. [...] Only 13% preferred the AI oral format. 57% wanted traditional written exams. [...] 83% of students found the oral exam framework more stressful than a written exam. [...] > Take-home exams are dead. Reverting to pen-and-paper exams in the classroom feels like a regression. Yeah, not sure the conclusion of the article really matches the…

I feel like the arms race between student cheaters and teacher testing has been going on for hundreds of years, ever since the first answer key written on the back of a hand

Re: Fighting Fire with Fire: Scalable Oral Exams

#230

Post by tracnar »

michaelt wrote: post #219
bmandale wrote: post #212

Earlier quoted context omitted.

I believe certain european countries have or had free universities which instead filter students with incredibly difficult courses. Thousands might enter because both tuition and board are free and they would like a degree, but the university ensures that only a small group make it to second year. I believe the filtering is less intense in later years, since the job has already been done by that point.

Unless you're thinking of huge online courses like Udacity/Coursera, I don't think that's really a thing? If it is, I'd be fascinated to learn more. I mean, the logistics would be pretty wild - even a large university's largest lecture theatres might only have 500 seats. And they'd only have one or two that large. It'd be expensive as hell to build a university that could handle multiple subjects each admitting over…

At least in Belgium it's quite common for a lot of students to fail the first year (partly due to the difficulty, partly due to partying instead of studying). But it's not like it's really free, the tuition is cheap but the accomodation is expensive. I also don't think it's particularly difficult on purpose to filter out students, it's just that it's not overly expensive and a lot of people are unsure about what to study.
Post reply on HN