Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Do AI detectors work? Students face false cheating accusations

bloomberg.com

Post reply on HN
Print view View original

391–400 of 1001 posts

Re: Do AI detectors work? Students face false cheating accusations

#391

Post by potato3732842 »

light_hue_1 wrote: post #277
michaelt wrote: post #254

Earlier quoted context omitted.

> Such as? We need an answer now because students are being assessed now. Two decades ago, when I was in engineering school, grades were 90% based on in-person, proctored, handwritten exams. So assignments had enough weight to be worth completing, but little enough that if someone cheated, it didn't really matter as the exam was the deciding factor. > Return to the old "viva voce" exam? Still used for PhDs. But that…

Oh yes. When I'm teaching a class of 200 students it's totally plausible that we're going to do 10 15 minute one on one conversations with every student. Because that's only 20 days non stop with no sleep. We would need to increase the amount of teaching staff by well over 10x to do this. The costs would be astronomical.

>We would need to increase the amount of teaching staff by well over 10x to do this. The costs would be astronomical.

We all know they'll just exploit grad students rather than hire real teachers.

Re: Do AI detectors work? Students face false cheating accusations

#392

Post by tonypace »

sumo89 wrote: post #188
greatartiste wrote: post #111

For a human who deals with student work or reads job applications spotting AI generated work quickly becomes trivially easy. Text seems to use the same general framework (although words are swapped around) also we see what I call 'word of the week' where whichever 'AI' engine seems to get hung up on a particular English word which is often an unusual one and uses it at every opportunity. It isn't long before you real…

My other half is a non-native English speaker. She's fluent but and since ChatGPT came out she's found it very helpful having somewhere to paste a paragraph and get a better version back rather than asking me to rewrite things. That said, she'll often message me with some text and I've got a 100% hit rate for guessing if she's put it through AI first. Once you're used to how they structure sentences it's very easy to…

It looked like black magic at first. But then you started to see the signs.

Re: Do AI detectors work? Students face false cheating accusations

#393

Post by batch12 »

light_hue_1 wrote: post #277
michaelt wrote: post #254

Earlier quoted context omitted.

> Such as? We need an answer now because students are being assessed now. Two decades ago, when I was in engineering school, grades were 90% based on in-person, proctored, handwritten exams. So assignments had enough weight to be worth completing, but little enough that if someone cheated, it didn't really matter as the exam was the deciding factor. > Return to the old "viva voce" exam? Still used for PhDs. But that…

Oh yes. When I'm teaching a class of 200 students it's totally plausible that we're going to do 10 15 minute one on one conversations with every student. Because that's only 20 days non stop with no sleep. We would need to increase the amount of teaching staff by well over 10x to do this. The costs would be astronomical.

200 students at 15 minutes is 50 hours or 33 hours and 20 minutes with 10 minute sessions. So just around the amount of time in a typical work week.

Re: Do AI detectors work? Students face false cheating accusations

#394

Post by jvanderbot »

gklitz wrote: post #330
lwhi wrote: post #178

It is no longer effective to solely use a written essay to measure how deeply a student comprehends a subject. AI is here to stay; new methods should be used to assess student performance. I remember being told at school, that we weren't allowed to use calculators in exams. The line provided by teachers was that we could never rely on having a calculator when we need it most—obviously there's irony associated with ha…

Written assay evaluation is not and has never been an effective evaluation. It was always a cost saving measure because allocating 30min face to face time with each individual student for each class is such a gigantic cost for the institution that they cannot even imagine doing it. Think about that the next time you look at your student debt, it couldn’t even buy you 30min time per class individually with the teacher…

Well, that's not necessarily true. I was perhaps the most importunate student ever, and lingered around my professor's offices whenever they were open. I had endless questions, off topic and on. I was curious sure, but I was also annoying and pushy and wouldn't take no for an answer.

In fact, the only reason I use the word 'importunate' to describe myself, is because that's what my undergrad advisor called me.

So I at least was able to get well over 30m with each professor to discuss whatever I wanted. But likely that's b/c there wasn't a lot of competition.

Re: Do AI detectors work? Students face false cheating accusations

#395

Post by fuzzy_biscuit »

If AI detection cannot be 100% accurate, I do not believe it is an appropriate solution for judging the futures of millions of students and young people. Time to move on. Either from the tech or from the essay format.

In either case, we need to change our standards around mastery of subject matter.

Re: Do AI detectors work? Students face false cheating accusations

#396

Post by darepublic »

The teacher will have to know the students better I suppose. If suspected of cheating a student should be given the chance to provide a writing sample and demonstrate knowledge in a face to face interview. It seems like the assignment here was just a summarization of some body of knowledge being taught; not something requiring original insight. It is the original ideas and analysis that, at this point, seems out of the reach of ai. And I guess more assessments of students should be testing the ability to produce such original insight, rather than what amounts to just a test or comprehension.

Re: Do AI detectors work? Students face false cheating accusations

#398

Post by teeray »

jmugan wrote: post #18

My daughter was accused of turning in an essay written by AI because the school software at her online school said so. Her mom watched her write the essay. I thought it was common knowledge that it was impossible to tell whether text was generated by AI. Evidently, the software vendors are either ignorant or are lying, and school administrators are believing them.

> I thought it was common knowledge that it was impossible to tell whether text was generated by AI.

I think it is, however the dream among educators of an “AI detector” is so strong that they’re willing to believe “these guys are the ones that cracked the problem” over and over, when it’s not entirely true. They try it out themselves with some simple attempts and find that it mostly works and conclude the company’s claims are true. The problem though is that their tests are all trying to pass off AI-generated work as human-generated—not the other way around. Since these tools have a non-zero false positive rate, there will always exist some poor kid who slaved away on a 20-page term paper for weeks that gets popped for using AI. That kid has no recourse, no appeals—the school spent a lot of money on the AI detector, and you better believe that it’s right.

Re: Do AI detectors work? Students face false cheating accusations

#399

Post by kenjackson »

tonypace wrote: post #336
strogonoff wrote: post #201

Earlier quoted context omitted.

The best method for assessing performance when learning is as old as the world: assess the effort, not how well the result complies with some requirements. If the level of effort made is high, but the outcome does not comply in some way, praise is due. If the outcome complies, but the level of effort is low, there is no reason for praise (what are you praising? mere compliance?) and you must have set a wrong bar. Not…

It's fairly simple in most situations. If it doesn't involve a computer, it's handwritten in class. If it does involve a computer, it's a temporarily offline computer. We have figured out solutions to these problems already.

It may be that offline LLMs will be common in a few years.

Re: Do AI detectors work? Students face false cheating accusations

#400

Post by hnpolicestate »

These policy questions are framed to always have the authoritarian win.

It's because AI detectors don't have 100% accuracy that they are considered bad.

Working AI detectors are bad.

False cheating accusations are collateral damage to those naive enough to participate.

Post reply on HN