Do AI detectors work? Students face false cheating accusations • Hacker News Board

Re: Do AI detectors work? Students face false cheating accusations

#261

Post by jenscow » Mon, Oct 21, 2024, 9:35 AM UTC

dambi0 wrote: post #214

lwhi wrote: post #178

It is no longer effective to solely use a written essay to measure how deeply a student comprehends a subject. AI is here to stay; new methods should be used to assess student performance. I remember being told at school, that we weren't allowed to use calculators in exams. The line provided by teachers was that we could never rely on having a calculator when we need it most—obviously there's irony associated with ha…

An essay written under examination conditions is fine. We don’t need new assessment techniques. We have known how to asses that a student and that student alone for centuries.

In most cases that only tests a students memory and handwriting ability, while under pressure in a limited time.

Can't perform any research, compare conflicting sources, or self-reflection.

Re: Do AI detectors work? Students face false cheating accusations

#262

Post by zelphirkalt » Mon, Oct 21, 2024, 9:35 AM UTC

borski wrote: post #28

ameister14 wrote: post #20

The article mentions 'responsible' grammarly usage, which I think is an oxymoron in an undergraduate or high school setting. Undergrad and high school is where you learn to write coherently. Grammarly is a tool that actively works against that goal because it doesn't train students to fix the grammatical mistakes, it just fixes it for them and they become steadily worse (and less detail oriented) writers. I have abso…

My wife is dyslexic; grammarly makes suggestions, but it doesn’t fix it for her. Perhaps that’s a feature she doesn’t have turned on? She loves it. It doesn’t cause her to be any less attentive to her writing; it just makes it possible to write.

An alternative idea could be to use some software that does speech to text. Not sure there are any easy to setup local options. I tried one a while ago, but not really investing much time into it, like some people do, who program using such a setup. The result was very underwhelming. Punctuation worked badly and capitalization of words also was non-existent, which of course would be a no-go for writing research papers.

So if anyone knows a good tool, that is flexible enough to support proper writing and able to run locally on a machine, hints appreciated.

Re: Do AI detectors work? Students face false cheating accusations

#263

Post by another-dave » Mon, Oct 21, 2024, 9:35 AM UTC

pjc50 wrote: post #205

lwhi wrote: post #178

It is no longer effective to solely use a written essay to measure how deeply a student comprehends a subject. AI is here to stay; new methods should be used to assess student performance. I remember being told at school, that we weren't allowed to use calculators in exams. The line provided by teachers was that we could never rely on having a calculator when we need it most—obviously there's irony associated with ha…

> I only hope that we get to decide how society responds to that change together .. rather than have it forced upon us. That basically never happens and the outcome is the result of some sort of struggle. Usually just a peaceful one in the courts and legislatures and markets, but a struggle nonetheless. > new methods should be used to assess student performance. Such as? We need an answer now because students are bei…

> Such as? We need an answer now because students are being assessed now.

When I was in university (Humanities degree), we had to do lots of mandatory essays throughout the year but they counted little towards your overall mark, maybe 10% iirc.

The majority of marks came from mid-year & end-of-year exams.

A simple change to negate AI is to not award any points for work outside exams — make it an optional chance to get feedback from lecturers. If students want to turn in work by AI, it's up to them

Re: Do AI detectors work? Students face false cheating accusations

#264

Post by jachee » Mon, Oct 21, 2024, 9:37 AM UTC

JoshTriplett wrote: post #204

greatartiste wrote: post #111

For a human who deals with student work or reads job applications spotting AI generated work quickly becomes trivially easy. Text seems to use the same general framework (although words are swapped around) also we see what I call 'word of the week' where whichever 'AI' engine seems to get hung up on a particular English word which is often an unusual one and uses it at every opportunity. It isn't long before you real…

> also we see what I call 'word of the week' where whichever 'AI' engine seems to get hung up on a particular English word which is often an unusual one and uses it at every opportunity So do humans. Many people have pet phrases or words that they use unusually often compared to others.

In the mid 90s (yes I’m dating myself here. :P) I had a classmate who was such a big NIN fan that she worked the phrase “downward spiral” into every single essay she wrote for the entire year.

Re: Do AI detectors work? Students face false cheating accusations

#265

Post by dambi0 » Mon, Oct 21, 2024, 9:38 AM UTC

jenscow wrote: post #261

dambi0 wrote: post #214

Earlier quoted context omitted.

An essay written under examination conditions is fine. We don’t need new assessment techniques. We have known how to asses that a student and that student alone for centuries.

In most cases that only tests a students memory and handwriting ability, while under pressure in a limited time. Can't perform any research, compare conflicting sources, or self-reflection.

That depends on the questions. There are also open book exams. A viva is a type of exam so I don’t see they are incompatible with assessing research

Re: Do AI detectors work? Students face false cheating accusations

#266

Post by mrweasel » Mon, Oct 21, 2024, 9:38 AM UTC

ben_w wrote: post #219

mrweasel wrote: post #184

The part that annoys me is that students apparently have no right to be told why the AI flagged their work. For any process where an computer is allowed to judge people, where should be a rule in place that demands that the algorithm be able explains EXACTLY why it flagged this person. Now this would effectively kill off the current AI powered solution, because they have no way of explaining, or even understanding, w…

> For any process where an computer is allowed to judge people, where should be a rule in place that demands that the algorithm be able explains EXACTLY why it flagged this person. This is a big part of GDPR.

I did not know that. Thank you.

Reading the rules quickly, it does seem like you're not entitled to know why the computer flagged you, only that you have the right to "obtain human intervention". That seems a little to soft, I'd like to know under which rules exactly I'm being judged.

Re: Do AI detectors work? Students face false cheating accusations

#267

Post by lwhi » Mon, Oct 21, 2024, 9:40 AM UTC

pjc50 wrote: post #205

lwhi wrote: post #178

It is no longer effective to solely use a written essay to measure how deeply a student comprehends a subject. AI is here to stay; new methods should be used to assess student performance. I remember being told at school, that we weren't allowed to use calculators in exams. The line provided by teachers was that we could never rely on having a calculator when we need it most—obviously there's irony associated with ha…

> I only hope that we get to decide how society responds to that change together .. rather than have it forced upon us. That basically never happens and the outcome is the result of some sort of struggle. Usually just a peaceful one in the courts and legislatures and markets, but a struggle nonetheless. > new methods should be used to assess student performance. Such as? We need an answer now because students are bei…

> Personally I think all this is unpredictable and destabilizing.

I completely agree, but then again it seems to me that society also functions according to many norms that were established due to historical context; and could / should be challenged and replaced.

Our education system was based on needs of the industrial revolution. Ditto, the structure of our working week.

My bet: We will see our working / waking lives shift before our eyes, in a manner that's comparable to watching an avalanche in the far distance. And (similarly to the avalanche metaphor) we'll likely have little ability to effect any change.

Fundamental questions like 'why do we work', 'what do we need' and 'what do we want' will be necessarily brought to the fore.

Re: Do AI detectors work? Students face false cheating accusations

#268

Post by anavat » Mon, Oct 21, 2024, 9:43 AM UTC

pjc50 wrote: post #205

lwhi wrote: post #178

It is no longer effective to solely use a written essay to measure how deeply a student comprehends a subject. AI is here to stay; new methods should be used to assess student performance. I remember being told at school, that we weren't allowed to use calculators in exams. The line provided by teachers was that we could never rely on having a calculator when we need it most—obviously there's irony associated with ha…

> I only hope that we get to decide how society responds to that change together .. rather than have it forced upon us. That basically never happens and the outcome is the result of some sort of struggle. Usually just a peaceful one in the courts and legislatures and markets, but a struggle nonetheless. > new methods should be used to assess student performance. Such as? We need an answer now because students are bei…

> But that doesn't scale at all.

It doesn't scale if performed by a human. But what if... we employ AI to conduct the voice exams?

Re: Do AI detectors work? Students face false cheating accusations

#269

Post by strogonoff » Mon, Oct 21, 2024, 9:44 AM UTC

injidup wrote: post #239

strogonoff wrote: post #201

Earlier quoted context omitted.

The best method for assessing performance when learning is as old as the world: assess the effort, not how well the result complies with some requirements. If the level of effort made is high, but the outcome does not comply in some way, praise is due. If the outcome complies, but the level of effort is low, there is no reason for praise (what are you praising? mere compliance?) and you must have set a wrong bar. Not…

Little Johnny who tried really hard but still can barely write a for loop doesn't deserve a place in a comp sci course ahead of little Timmy who for some reason thinks in computer code. Timmy might be a lazy arse but he's good at what he does and for minimal effort the outcomes are amazing. Johnny unfortunately just doesn't get it. He's wanted to be a programmer ever since he saw the movie Hackers but his brain just…

My evaluation:

1. Whoever determined that he does not “deserve” this is wrong. There may be other constraints, but no one gets to frame it as “deserves” when a child wants to learn something.

2. If a teacher is unable to teach Johnny to write a for loop, despite Johnny’s genuine utmost motivation, I would question teacher’s competence or at least fit.

3. Like any mentor, a professor in higher ed may want to choose whom to teach so that own expertise and teaching ability is realized to the fullest. Earlier in life, elementary school teacher’s luxury to do so may be limited (which is why their job is so difficult and hopefully well-compensated), and one bailing on a kid due to lack of patience or teaching competence is detestable.

4. If Johnny continues to pursue this with genuine utmost motivation, he will most likely succeed despite any incompetent teachers. If he does not succeed and yet continues to pursue this to the detriment to his life, that is something a psychologist should help him with.

As for Timmy, if he learns to produce the expected result with least effort, for which he receives constant praise from the teacher, and keeps coasting this way, that does him a major disservice as far as mental mental and self-actualisation in life.

Re: Do AI detectors work? Students face false cheating accusations

#270

Post by strogonoff » Mon, Oct 21, 2024, 9:46 AM UTC

concordDance wrote: post #225

strogonoff wrote: post #201

Earlier quoted context omitted.

The best method for assessing performance when learning is as old as the world: assess the effort, not how well the result complies with some requirements. If the level of effort made is high, but the outcome does not comply in some way, praise is due. If the outcome complies, but the level of effort is low, there is no reason for praise (what are you praising? mere compliance?) and you must have set a wrong bar. Not…

> The best method for assessing performance when learning is as old as the world: assess the effort, not how well the result complies with some requirements. I am really quite confused about what you think the point of education is. In general, the world (either the physical world or the employment world) does not care about effort, it cares about results. Someone laboriously filling their kettle with a teaspoon migh…

You may be mistaking “the world” with “education” or “learning”. Producing a result is not evidence of learning progress. During learning, result is a somewhat useful metric if it roughly correlates with the level of effort, but relying only on result when determining whether to praise or reward a person during the learning stage is always a recipe for issues. A student may quickly learn to reproduce the desired result and stop progressing.