> 36 students examined over 9 days, 25 minutes average I could accept this for a 300 students class, but 36? When I got my degree, ALL exams had an oral component, usually more than 30 minutes long. The prof and one or two TAs would take a couple days and just do it. For 36 students it’s more than doable. If I was a student being examined by an LLM I would feel like the professor didn’t care enough to do the work.
In general when you try a new tool or methodology you tend to start with a small class to see the results first.
Lots of emotional commenting here. This guy, Panos Ipeirotis, is seriously on to the way university testing and corporate seminar testing will be done in the immediate future, as well as going forward. Complain all you want, this is inevitable. This initial version will improve. In time, more complex and multi-mod voice agents will do the teaching too, entirely individualized as well.
Did you make it far enough to find out about his "Docent" system for AI exams? If it's not a startup yet, he's thinking about it. [1]: https://get-docent.com/
I’m doubtful of most of the “fixes”. Putting more instructions in the prompt can maybe make the LLM more likely to follow them, but it’s by no means guaranteed.
I basically agree with the thrust of what you're saying, but also: > I wrote assembly and C++ code by hand, and it was expected to compile. No, I never got a chance to try to compile it myself before submitting it for grading. Do you, like, really think this is the best way to assess someone's ability? Can't we find a place between the two extremes? Personally, I'd go with a school-provided computer with a developmen…
The safe middle space still does not involve a computer Lots of my tests involved writing pseudocode, or "Just write something that looks like C or Java". Don't miss the semicolon at the end of the line, but if you write "System.print()" rather than "System.out.printLn()" you might lose a single point. Maybe. If there were specific functions you need to call, it would have a man page or similar on the test itself, or…
In my CS curriculum we learned SQL in theory only. We learned the relational model, normalization, joins, predicates, aggregation, etc. all without ever touching an actual database. In the exams we wrote queries in a paper "blue book" which was graded by teaching assistants.
At the risk of perhaps stating the obvious, there appears to be a whiff of aggression from this article. The "fighting fire with fire" language, the "haha, we love old FakeFoster, going to have to see if we change that" response to complaints that the voice was intimidating ... if there wasn't a specific desire to punish the class for LLM use by subjecting them to a robotic NKVD interrogation then the authors should…
The belligerence about changing the voice is so weird. And it does sort of set a tone straight off. "We got feedback that the voice was frightening and intimidating. We're keeping it tho."
It’s not an intimidating voice. Gen Z are just cry babies.
This is all so crazy to me. I went to school long before LLMs were even a Google Engineer's brianfart for the transformer paper and the way I took exams was already AI proof. Everything hand written in pen in a proctored gymnasium. No open books. No computers or smart phones, especially ones connected to the internet. Just a department sanctioned calculator for math classes. I wrote assembly and C++ code by hand, and…
I'm fairly skeptical of tests that are closed-book. IMO the only reasons to do so are if 1) the goal is to test rote memorization (which is admittedly sometimes valuable, especially depending on the field) or, perhaps more commonly, 2) the test isn't actually hard enough, and the questions don't require as much "synthesis" as they should to test real understanding.
At the risk of perhaps stating the obvious, there appears to be a whiff of aggression from this article. The "fighting fire with fire" language, the "haha, we love old FakeFoster, going to have to see if we change that" response to complaints that the voice was intimidating ... if there wasn't a specific desire to punish the class for LLM use by subjecting them to a robotic NKVD interrogation then the authors should…
You can try out the voice yourself. It's not that bad. https://elevenlabs.io/app/talk-to?agent_id=agent_8101k9d1pq4...
Tried it in earnest. Definitely detect some aggression, and would feel stressed if this were an exam setting. I think it was pg who said that any stress you add in an interview situation is just noise, and dilutes the signal.
Also, given that there's so many ways for LLMs to go off the rails (it just gave me the student id I was supposed to say, for example), it feels a bit unprofessional to be using this to administer real exams.