Do AI detectors work? Students face false cheating accusations • Hacker News Board

Re: Do AI detectors work? Students face false cheating accusations

#531

Post by WillAdams » Mon, Oct 21, 2024, 2:12 PM UTC

gklitz wrote: post #330

lwhi wrote: post #178

It is no longer effective to solely use a written essay to measure how deeply a student comprehends a subject. AI is here to stay; new methods should be used to assess student performance. I remember being told at school, that we weren't allowed to use calculators in exams. The line provided by teachers was that we could never rely on having a calculator when we need it most—obviously there's irony associated with ha…

Written assay evaluation is not and has never been an effective evaluation. It was always a cost saving measure because allocating 30min face to face time with each individual student for each class is such a gigantic cost for the institution that they cannot even imagine doing it. Think about that the next time you look at your student debt, it couldn’t even buy you 30min time per class individually with the teacher…

My best college professor (who was also an Episcopalian Priest) found the time to review one paper with each student once per semester.

That strikes me as a workable bottom line.

Re: Do AI detectors work? Students face false cheating accusations

#532

Post by bonoboTP » Mon, Oct 21, 2024, 2:16 PM UTC

lwhi wrote: post #178

It is no longer effective to solely use a written essay to measure how deeply a student comprehends a subject. AI is here to stay; new methods should be used to assess student performance. I remember being told at school, that we weren't allowed to use calculators in exams. The line provided by teachers was that we could never rely on having a calculator when we need it most—obviously there's irony associated with ha…

Learning takes time. And the fully trained/educated/skilled/expert human performance is higher than AI performance. But AI performance may be higher than intermediate human performance after 1 or 2 semesters. But you need to reach intermediate performance first in order to later reach expert performance. During that time you still need a learning "slope", you need to be tested on your knowledge at that level. If you're given the AI at the outset, you will not develop the skill to surpass the AI performance.

Calculators are just one analogy, there is no guarantee it will work out that way. It's just as likely that this over-technologization of the classroom will go the way of whole-language reading education.

Re: Do AI detectors work? Students face false cheating accusations

#533

Post by bearjaws » Mon, Oct 21, 2024, 2:16 PM UTC

Wheatman wrote: post #513

bearjaws wrote: post #469

Earlier quoted context omitted.

I recently deployed an AI detector for a large K12 platform (multi-state 20k+ students), and they DO work in the sense of saving teachers time. You have to understand, you are a smart professional individual who will try to avoid being detected, but 6-12th grade students can be incredibly lazy and procrastinate. You may take the time to add a tone, style and cadence to your prompt but many students do not. They can b…

Doesnt google docs have a feature that shows writing history. You could ask the student to start wrkting on google docs, and whenever someone gets a false positive, they can prove they wrote it through that. And Besides 99% of people who use AI to write, dont bother claiming it as a false positive, so giving students the right to contest that claim would not be that much if a problem long term.

Yeah, those are great points, and our students do use Google Docs today, and you are right most students do not even contest it.

We let them resubmit a new paper when they are caught, and they get some one on one time with a tutor to help move them forward. Typically they were stuck or rushing, which is why they dumped a whole AI slop assignment into our LMS.

Re: Do AI detectors work? Students face false cheating accusations

#534

Post by Izkata » Mon, Oct 21, 2024, 2:18 PM UTC

obscurette wrote: post #408

delusional wrote: post #199

Earlier quoted context omitted.

> I remember being told at school, that we weren't allowed to use calculators in exams I remember being told the same thing, but I happen to believe that it was a fantastic policy, with a lackluster explanation. The idea that you wouldn't have a calculator was obviously silly, even at the time, but underlying observation that relying on the calculator would rob you of the mental exercise the whole ordeal was supposed…

It's actually not about beauty of the math, it's about something which is nowadays called a number sense. It takes a lot of practice to develop an understanding what these things called numbers are, how these relate to each other, what happens if you combine these with operational signs, how numbers grow and shrink etc. And you are damn right that there is no any use to explain it to the 12 year olds. Or even to 16 y…

Common Core was an attempt at teaching this directly. It gets so much ridicule because so few people have good enough number sense to recognize what they're seeing when shown a demonstration. Of course, since they didn't understand it, it then led to bad examples being created and shared, which just made it worse...

Re: Do AI detectors work? Students face false cheating accusations

#535

Post by skhunted » Mon, Oct 21, 2024, 2:19 PM UTC

bonoboTP wrote: post #527

skhunted wrote: post #491

I’ve been teaching in higher education for 30 years and am soon retiring. I teach math. In every math course there is massive amounts of cheating on everything that is graded that is not proctored in a classroom setting. Locking down browsers and whatnot does not prevent cheating. The only solution is to require face-to-face proctored exams and not allow students to use technology of any kind while taking the test. B…

> The only solution is to require face-to-face proctored exams and not allow students to use technology of any kind while taking the test. In Germany, all exams are like this. Homework assignments are either just a prerequisite for taking exam but the grade is solely from the exam, or you may get some small point bonus for assignments/projects. > But any teacher doing this will end up with no students signing up for…

You can't "not sign up" for linear algebra if it's in your curriculum.

Same in the U.S. but you can sometimes find an online offering. If you don’t know what you are doing or don’t care then always take the online offering. Much easier to cheat.

My ex-girlfriend is German. She cheated on her exams to get her agricultural engineering degree at university. This was in the 80s.

Re: Do AI detectors work? Students face false cheating accusations

#536

Post by bonoboTP » Mon, Oct 21, 2024, 2:21 PM UTC

red_admiral wrote: post #278

lwhi wrote: post #178

It is no longer effective to solely use a written essay to measure how deeply a student comprehends a subject. AI is here to stay; new methods should be used to assess student performance. I remember being told at school, that we weren't allowed to use calculators in exams. The line provided by teachers was that we could never rely on having a calculator when we need it most—obviously there's irony associated with ha…

> It is no longer effective to solely use a written essay to measure how deeply a student comprehends a subject. It never was. It's just even more ineffective now that AI exists, than before. The central example of this is college admissions statements. Some kids have the advantage both of parents who can afford to give them the experiences that look good on such an essay (educational trips to Africa, lessons in two…

> The central example of this is college admissions statements. Some kids have the advantage both of parents who can afford to give them the experiences that look good on such an essay (educational trips to Africa, lessons in two musical instruments, one-on-one golf coaching, that kind of thing), and who can hire tutors to "support" them in writing the essay.

This is an absolute disgrace. And then these are the people who lecture you on "inclusion".

Re: Do AI detectors work? Students face false cheating accusations

#537

Post by lazide » Mon, Oct 21, 2024, 2:21 PM UTC

SkyBelow wrote: post #515

skhunted wrote: post #491

I’ve been teaching in higher education for 30 years and am soon retiring. I teach math. In every math course there is massive amounts of cheating on everything that is graded that is not proctored in a classroom setting. Locking down browsers and whatnot does not prevent cheating. The only solution is to require face-to-face proctored exams and not allow students to use technology of any kind while taking the test. B…

>If we did truly devise a system that prevents cheating we’ll see that a very high percentage of current college students are not ready to be truly college educated. Isn't it to either do that now, or to lose the signaling value of college degrees as indicating knowledge.

That signaling value was lost years ago.

Re: Do AI detectors work? Students face false cheating accusations

#538

Post by yojo » Mon, Oct 21, 2024, 2:22 PM UTC

lwhi wrote: post #178

It is no longer effective to solely use a written essay to measure how deeply a student comprehends a subject. AI is here to stay; new methods should be used to assess student performance. I remember being told at school, that we weren't allowed to use calculators in exams. The line provided by teachers was that we could never rely on having a calculator when we need it most—obviously there's irony associated with ha…

I won’t claim this is by design, but at the very least a side effect of writing term papers is getting practice at organizing your thoughts and drawing conclusions from them.

While writing term papers is a skill that is only minimally useful in the real world (save for grant writers and post docs, pretty much), the patterns of thinking it encourages are valuable to everything that isn’t ditch digging.

Maybe we can outsource this part of our cognition to AI, but I’m skeptical of the wisdom of doing so. Are we all going to break to consult ChatGPT in strategy meetings?

Re: Do AI detectors work? Students face false cheating accusations

#539

Post by _heimdall » Mon, Oct 21, 2024, 2:23 PM UTC

smartmic wrote: post #517

_heimdall wrote: post #372

Earlier quoted context omitted.

This feels like an argument bigger than AI evaluations. All points you raised could very well be issues with humans evaluating other humans to attempt to predict future outcomes.

They are not wrong. And the art of predicting future outcomes proves to be difficult and fraught with failure. But human evaluation of other humans is more like an open level field to me. A human is accountable for what he or she says or predicts about others, subject to interrogation or social or legal consequences. Not so easy with AI, because it steps out of all these areas - at least many actors using AI do not s…

In my experience, we're really bad at holding humans accountable for their predictions too. That may even be a good thing, but I'm less confident that we would be holding LLMs less accountable for their predictions than humans.

Re: Do AI detectors work? Students face false cheating accusations

#540

Post by pdonis » Mon, Oct 21, 2024, 2:23 PM UTC

SkyBelow wrote: post #515

skhunted wrote: post #491

I’ve been teaching in higher education for 30 years and am soon retiring. I teach math. In every math course there is massive amounts of cheating on everything that is graded that is not proctored in a classroom setting. Locking down browsers and whatnot does not prevent cheating. The only solution is to require face-to-face proctored exams and not allow students to use technology of any kind while taking the test. B…

>If we did truly devise a system that prevents cheating we’ll see that a very high percentage of current college students are not ready to be truly college educated. Isn't it to either do that now, or to lose the signaling value of college degrees as indicating knowledge.

> the signaling value of college degrees as indicating knowledge

I'm not sure knowledge is what a college degree signals to prospective employers. The alternative hypothesis, which AFAIK has a fair bit of support, is that it signals a willingness to do whatever it takes to fulfill a set of on paper requirements imposed by an institution, by hook or by crook.