Do AI detectors work? Students face false cheating accusations • Hacker News Board

Re: Do AI detectors work? Students face false cheating accusations

#521

Post by bearjaws » Mon, Oct 21, 2024, 1:59 PM UTC

max51 wrote: post #507

bdzr wrote: post #460

Earlier quoted context omitted.

What solutions are 100% accurate?

The problem is that AI detection is far closer to 0% than 100%,. It's really bad and the very nature of this tech makes it impossible to be good.

As someone working in this field, it is simply not closer to 0%

People keep using these "gotcha" examples and never actually look at the stats for it. I get it, there are some terrible detectors out there, and of course they are the free ones :)

https://edintegrity.biomedcentral.com/articles/10.1007/s4097...

GPTZero was correct in most scenarios where they used basic prompts, and only had one false positive.

We did a comparison of hand reviewed 3,000 9-12th grade assignments and found that GPTZero holds up really well.

In the same way that plagiarism detectors need a process for review, your educational institution needs the same for AI detection. Students shouldn't be immediately punished, but instead it should be reviewed, and then an appropriate decision made by a person.

Re: Do AI detectors work? Students face false cheating accusations

#522

Post by elric » Mon, Oct 21, 2024, 2:01 PM UTC

everdrive wrote: post #302

elric wrote: post #260

Earlier quoted context omitted.

Was this ever effective? There was a lot of essay copy/pasting when I was in school, and this was when essays had to be hand written (in cursive, of course, using a fountain pen!). Same with homework. If everyone has to solve the same 10 problems, divide and conquer saves everyone a lot of time. Of course, you're only screwing yourself because you'll negatively impact your learning, but that's not something you can e…

>Of course, you're only screwing yourself because you'll negatively impact your learning, but that's not something you can easily convince kids of. This assumes that homework helps kids learn, or that the knowledge required to succeed in school will help kids once they graduate.

Depends on the homework, of course. In my head I guess I was talking about maths problems. Maths understanding, in my experience, greatly benefits from practice, and homework exercises might be useful there. Memorising the names of rivers ... maybe not so much.

Re: Do AI detectors work? Students face false cheating accusations

#523

Post by mensetmanusman » Mon, Oct 21, 2024, 2:01 PM UTC

tdeck wrote: post #80

mensetmanusman wrote: post #3

My daughter’s 7th grade work is 80% flagged as AI. She is a very good writer, it’s interesting to see how poorly this will go. Obviously we will go back to in class writing.

> Obviously we will go back to in class writing. That would be a pretty sad outcome. In my high school we did both in-class essays and homework essays. The former were always more poorly developed and more more poorly written. IMO students still deserve practice doing something that takes more than 45 minutes.

Could be a Saturday event in a comfortable setting. People can still practice, but then will have to somehow prove they aren’t AI :)

Re: Do AI detectors work? Students face false cheating accusations

#524

Post by skhunted » Mon, Oct 21, 2024, 2:02 PM UTC

SkyBelow wrote: post #515

skhunted wrote: post #491

I’ve been teaching in higher education for 30 years and am soon retiring. I teach math. In every math course there is massive amounts of cheating on everything that is graded that is not proctored in a classroom setting. Locking down browsers and whatnot does not prevent cheating. The only solution is to require face-to-face proctored exams and not allow students to use technology of any kind while taking the test. B…

>If we did truly devise a system that prevents cheating we’ll see that a very high percentage of current college students are not ready to be truly college educated. Isn't it to either do that now, or to lose the signaling value of college degrees as indicating knowledge.

Yes. But people now teaching at higher education institutions need their classes to fill. That means we need to treat our students as if they are our customers. We must please the customer. In years past the attitude was that society at large was our client. Today the student is our client.

Re: Do AI detectors work? Students face false cheating accusations

#525

Post by michaelmrose » Mon, Oct 21, 2024, 2:05 PM UTC

ordu wrote: post #422

lukan wrote: post #376

Earlier quoted context omitted.

I hope that was sarcasm?

Probably. I'm not sure myself. It is, because I'm becoming tired with the current AI hype. It lasts too long to be funny. OTOH, professor talking with a student is a good way to assess the academic performance of the student, but there are some caveats beyond costs. For example, professor will struggle to be an objective judge. Moreover even if they succeed, they would face accusations of discrimination in any case.…

Actually deliberately introducing confidently delivered and reasonable sounding bullshit sounds like a fantastic way to suss out who knows their topic.

Re: Do AI detectors work? Students face false cheating accusations

#526

Post by eszed » Mon, Oct 21, 2024, 2:06 PM UTC

Teever wrote: post #338

gklitz wrote: post #330

Earlier quoted context omitted.

Written assay evaluation is not and has never been an effective evaluation. It was always a cost saving measure because allocating 30min face to face time with each individual student for each class is such a gigantic cost for the institution that they cannot even imagine doing it. Think about that the next time you look at your student debt, it couldn’t even buy you 30min time per class individually with the teacher…

There is truth to this perspective but it's also missing one of the fundamental purposes of writing essays in an educational setting. Writing essays isn't just about evaluation, it's also about teaching you how to think. The process of reading textual material, thinking about it, and then producing more textual material about the stuff you just read (and maybe connecting it to other stuff that you've read in the past…

You're completely correct. Learning how to write taught me how to think, and researching and writing essays taught me what I believe about nearly everything on which I have strong opinions.

However, +90% of students will not now do any of that work. I got out of teaching (coincidentally) before LLMs appeared, and even then +80% of students did not experience that benefit of the essay process even with a grade (and plagiarism consequences) to motivate them. Now that decent-ish prose is a few keystrokes or Siri-led "chats" away, that's what they're going to do. That's what they're going to do.

I know of - I think it's up to four, now - former colleagues taking early retirement, or changing careers, rather than continue teaching Humanities in a world of LLMs.

Re: Do AI detectors work? Students face false cheating accusations

#527

Post by bonoboTP » Mon, Oct 21, 2024, 2:06 PM UTC

skhunted wrote: post #491

I’ve been teaching in higher education for 30 years and am soon retiring. I teach math. In every math course there is massive amounts of cheating on everything that is graded that is not proctored in a classroom setting. Locking down browsers and whatnot does not prevent cheating. The only solution is to require face-to-face proctored exams and not allow students to use technology of any kind while taking the test. B…

> The only solution is to require face-to-face proctored exams and not allow students to use technology of any kind while taking the test.

In Germany, all exams are like this. Homework assignments are either just a prerequisite for taking exam but the grade is solely from the exam, or you may get some small point bonus for assignments/projects.

> But any teacher doing this will end up with no students signing up for their class.

The main courses are mandatory in order to obtain the degree. You can't "not sign up" for linear algebra if it's in your curriculum. Fail 3 times and you're exmatriculated.

This is because universities are paid from tax money in Germany and most of Europe.

The US will continue down on the path you describe because it's in the interest of colleges to keep well-paying students around. It's a service. You buy a degree, you are a customer.

Re: Do AI detectors work? Students face false cheating accusations

#528

Post by bearjaws » Mon, Oct 21, 2024, 2:07 PM UTC

Buttons840 wrote: post #490

Every student falsely accused of cheating will be one less ally teachers have when they start complaining about being replaced by AI.

HN and the tech community as a whole need to realize that education will actually still be one of the most hands-on jobs well into the future.

Students already have the entire internet at their disposal, the wealth of all human knowledge right in their hands, it's been over an entire internet generation at this point. I'd go as far as to argue many courses on YouTube are much higher quality than what they recieve in a classroom. Are students learning more than ever?

No, they are not, in fact many argue it's worse than ever, English comprehension and writing have regressed significantly.

Students often need support from their teachers, teachers often are more present than their parents. It simply isn't the case that most people will be a self-taught learner with an AI.

I do agree educators should pivot to more hands on, non-writing lessons such as debates, instead of written papers, but we're not going to improve writing skills without having written papers..

Re: Do AI detectors work? Students face false cheating accusations

#529

Post by Spivak » Mon, Oct 21, 2024, 2:07 PM UTC

jeroenhd wrote: post #519

washadjeffmad wrote: post #459

Earlier quoted context omitted.

https://news.ycombinator.com/item?id=41882421 My comment from a few days ago. The origin was a conversation with a girl who said she'd been pulled into a professor's office and told she was going to be reported to whatever her university's equivalent of Student Conduct and Academic Integrity is over using AI - a matter of academic honesty. The professor made it clear in the syllabus that "no AI" was allowed to be use…

In my opinion, it strongly depends on what Grammarly is being used for. For a physics paper, that's not a huge problem. For an English writing assignment, that's cheating. Banning AI tools like Grammarly for both is probably the best solution as your physics paper now becomes an extra training exercise for your English paper. Writing essays isn't just about your ideas. It's also a tool to teach communication skills.…

> For an English writing assignment, that's cheating

It's still not cheating. English assignments aren't about the practice of writing English, you stop doing that in primary school. It's analysis of English texts in which people have been using spelling and grammar checkers since their inception. It's not even cheating to have someone proofread and edit your paper, it's usually encouraged, and Grammarly is just a worse-than-human editor.

Re: Do AI detectors work? Students face false cheating accusations

#530

Post by jhbadger » Mon, Oct 21, 2024, 2:12 PM UTC

dagw wrote: post #489

thechao wrote: post #378

Earlier quoted context omitted.

If I'm paying 30k$/yr the professor is damn well reading my essay. If they don't want to teach & grade, they can get a pure research position. Fun fact: pure research positions don't pay as well.

Fun fact: pure research positions don't pay as well. Where do you get this from? The people I know with pure research positions get paid basically the same (after correcting for 'rank' and seniority) as those who split their time between research and teaching.

At least in the sciences, and in the US, there is also the issue that research professors tend to be on "soft money" -- that is they get a minimal salary from their institution but can increase it (up to a point) by getting grants that they can charge their time to. And they also tend not to be in the tenure track system. That being said, if they get large enough grants, they can make as much if not more than traditional tenure-track professors with defined salaries. But in years where they don't get much grant funding they don't make much at all (I used to be an non-tenure track research professor myself).