Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

College students are learning hard lessons about anti-cheating software

voiceofsandiego.org

Post reply on HN
Print view View original

51–60 of 317 posts

Re: College students are learning hard lessons about anti-cheating software

#51

Post by klodolph »

throwaway0a5e wrote: post #40
klodolph wrote: post #19

Earlier quoted context omitted.

This is an echo of the “guns don’t kill people, people kill people” argument that has been going around for ages. Let’s say you’re only using the software to flag suspicious behavior, and bringing in humans to make the final decision. What happens when (inevitably) the software disproportionally flags people with dark skin because it is not trained to recognize dark-skinned faces? Or when the software disproportional…

I don't think you meant to do this but you seem to have inadvertently made a very damn good argument for "guns don't kill people people kill people. People are getting screwed because they are at the long end of an unbroken chain of crap. Crappy organizations buy crappy software and crappy professors take the results seriously. The fact that there exists a crappy tool that flags all the black people as cheaters (or w…

This argument is fallacious. Your argument assumes that “blaming the gun” is condoning the users, and this is a false dilemma.

There’s really no room for absolutism, where blame is assigned to one source rather than distributed among many contributing factors. Imagine how dangerous air travel would still be if, after an accident, investigators looked for only one cause to blame, and tacitly condoned anything else they came across in their investigations.

Re: College students are learning hard lessons about anti-cheating software

#52

Post by iandanforth »

A few points:

Their "powerful AI engine" is almost certainly just humans. It might have a few off-the-shelf components like face detection but most of what they claim to do is just so easy to outsource that almost all companies do it. If there is any delay between the system observing a suspect behaviour and the student being told to correct it then they are definitely using humans.

An institution using a service like this is a huge red flag. You should take it as an indicator of a low quality administration if not a low quality institution.

As an engineering problem this task is hard. Ryan Calo (Prof of Law, UW) once presented a fascinating bit of research on trying to automate something as simple as fining someone for speeding. Given perfect information how do you build the system? If someone exceeds the speed limit for 1 second, do you fine them? If everyone around the person is exceeding the speed limit do you use the same rules? If someone oscillates between just above and just below the speed limit, how many times do you fine them? If someone exceeds the speed limit and stays there does this result in fewer fines? How do you square the code written with the law as written? The problems are so extensive it may be that application of rules like this require human level judgement. Proctoring an exam may turn out to be an AI-complete problem.

Re: College students are learning hard lessons about anti-cheating software

#53

Post by pdkl95 »

> At the self-described “heart” of the company’s monitoring software is Monitor AI, a “powerful artificial intelligence engine” that collects facial detection data [...] to identify “patterns and anomalies associated with cheating.”

> "... people who have some sort of facial disfigurement have special challenges; they might get flagged because their face has an unexpected geometry.”

So this company is extrapolating "patterns ... associated with cheating" from facial geometry. This is just phrenology laundered through their "powerful artificial intelligence engine" black box. Predicting behavior from the shape of someone's skull is still bullshit pseudoscience, even if the calipers[1] are replaced with a bunch of linear algebra.

[1] http://antiquescientifica.com/phrenology_calipers_George_Com...

Re: College students are learning hard lessons about anti-cheating software

#54

Post by RegnisGnaw »

TrackerFF wrote: post #42
serjester wrote: post #10

From the many students I know, I can tell you that cheating is rampant in online learning. The average GPA has gone up 0.3 points at my alma mater. I don't think people understand just how widespread it is. This software isn't ideal but there's no good solutions here short of major rethink of how these classes, and possible all of the college, is structured.

At least where I'm from (Norway), grade transcripts will show a class grade distribution/histogram in the background of your grade. That way the viewer / reader will be able to compare your results to your class. Obviously if 80% of your class got A, that A is not going to look as impressive. On the other hand, if you're the only student that got A, while the class distribution is monotonically increasing towards Fai…

When I went to college, our department gave two grades on the transcript: percentage mark and class ranking. The ranking is your ranking among all the sections of that class taught that semester.

Re: College students are learning hard lessons about anti-cheating software

#55

Post by hahamrfunnyguy »

serjester wrote: post #10

From the many students I know, I can tell you that cheating is rampant in online learning. The average GPA has gone up 0.3 points at my alma mater. I don't think people understand just how widespread it is. This software isn't ideal but there's no good solutions here short of major rethink of how these classes, and possible all of the college, is structured.

Cheating is rampant in in-person learning as well. We shouldn't be treating online learning the same as in-person learning. I took distance learning classes back in the early 00's. Some courses required proctored exams, some had open book exams. For proctored exams, I would do them in the library, then the librarian would send the test back to the school. The open book exams weren't really any easier - you were expected to know the material and therefore the tests were more challenging.

In my professional experience, GPA has little correlation with how well someone can do a job. GPA going up or down by 0.3 points really doesn't mean that much to me.

Re: College students are learning hard lessons about anti-cheating software

#56

Post by OldHand2018 »

huhnmonster wrote: post #36
wolfgke wrote: post #29

Earlier quoted context omitted.

Make exams that are harder to cheat. For example: - oral exams via video call - written exams where students are distributed over a larger area (e.g. the university rents a warehouse for the examination time) so that the COVID-19 spreading risk is nevertheless kept very small.

This has been done in my last exam phase and has worked well. They cranked the AC to 11, which made it quite an unpleasant envirommemt, but I would rather wear a jacket than take online exams

I can remember taking many exams in the athletic field house at my university back in the 1990s. It was pretty standard when you were in a large class.

The desks were so far apart that the current "social distancing" standards would be met. I remember that many courses had multiple variations of the same exam given to students to further reduce cheating.

I'm taking an online masters right now and have taken a few (pre-pandemic) proctored exams. The nearby university offers to proctor any exam for $10. They have a bunch of rooms with desks, and it's no big deal at all.

This proctoring technology thing has gone too far and too fast. It was a knee-jerk reaction, and with a lot of complaints I bet that a lot of it gets dropped.

Re: College students are learning hard lessons about anti-cheating software

#57

Post by parzivalm »

querez wrote: post #44
parzivalm wrote: post #25

The thing that has always frustrated me about anti-cheating/anti-plagiarism software is that it almost always only hurts people that did something by accident or unintentionally. When I took the one kubernetes certification exam as part of an old job, you used software like this and at one point I leaned too close to the screen and the proctor couldn't see my face and that got me flagged. People that want to cheat or…

> One of the issues is also just academia being so hyper-focused on thinking everyone is there to become an academic which is not the case for the vast majority of people I'm not sure I follow why this is bad. I think academic rigor and being a decent scholar, as well as being able to parse and produce research are good things (and in my mind, those are the corner stones of being an academic). Did misunderstand you?…

I don't think the problem is that academic rigor isn't good. I also should have stated I was talking more about US universities in this particular case.

The problem I see is that what you talk about as academic rigor isn't what is taught and evaluated in many of the programs and classes that I've seen or been a part of. A lot of these exams and assessments don't particularly evaluate you on your ability to research and understand knowledge. If I know for example that my physics professor uses a bank of questions then it is much more incentivized for me to memorize that bank of questions vs. understanding the content and working the problems myself. Whereas in say a Discrete Math or Algorithm based class, the final exam/grade is based on a proof you have to write yourself, that encourages (or rather at times forces) you to learn and research like you said.

I also think the issue, and this may just be me looking at from my own experience, a lot of people don't want to be scholars, as you put it. They went to a University based on the unfortunate expectation for some jobs that say you need that diploma as your entry ticket.

Re: College students are learning hard lessons about anti-cheating software

#58

Post by klodolph »

Terrario wrote: post #32
klodolph wrote: post #19

Earlier quoted context omitted.

This is an echo of the “guns don’t kill people, people kill people” argument that has been going around for ages. Let’s say you’re only using the software to flag suspicious behavior, and bringing in humans to make the final decision. What happens when (inevitably) the software disproportionally flags people with dark skin because it is not trained to recognize dark-skinned faces? Or when the software disproportional…

I think this misses the parent’s point. The point is to not let the algorithm make decisions. The human bureaucracy is suppose to be there to determine the quality of the flags and analyze whether there is any discrimination at play. A company that lacks this human element is negligent and should be held responsible. Unless algorithms can be trialed and held accountable, they shouldn’t be allowed to make decisions. A…

> I think this misses the parent’s point.

Just because you disagree with me, it doesn’t mean that I misunderstood the viewpoint I’m responding to.

> The point is to not let the algorithm make decisions.

And my response is—that’s not enough. It sounds like the algorithm, because it is biased, has the effect of increasing the bias in the whole system. If your response is that humans should work harder to counteract biases in machine systems, well, I think that’s just a way to CYA and assign blame but not a way to solve the problem—humans will remain biased, and they will trust automated systems even when that trust is misplaced.

As an analogy, it’s like a driver in a partially autonomous car. As soon as the automation takes over, the driver stops paying attention to the road. We can make a big fuss and production and talk about how it’s the driver’s fault, and the driver should pay attention, but we’ve placed them in a system where they are discouraged from paying attention, and the system is more dangerous as a consequence.

> Also guns don’t kill people, people do. Otherwise explain to me why it would be okay for certain institutions to be armed but not individuals. If guns are the problem, then no one should have them (including the military/police).

This is a false dilemma / false dichotomy. This argument assumes that EITHER access to guns is to blame OR people are to blame, but not both, but there are obviously other ways to think about the problem.

Any rational way to look at problems will look at multiple contributing factors.

Re: College students are learning hard lessons about anti-cheating software

#59

Post by jjk166 »

ryandrake wrote: post #33
jjk166 wrote: post #18

Earlier quoted context omitted.

What stopped people from having someone else do their take home tests before the pandemic? On a more constructive note, it seems pretty simple to require some portion of responses to reference the lectures such that only someone who attended the class in this semester would be able to correctly answer, for example "use the method we discussed in the first half of lecture 3, use the third of the four numbers I told yo…

> require some portion of responses to reference the lectures such that only someone who attended the class in this semester would be able to correctly answer, for example "use the method we discussed in the first half of lecture 3, use the third of the four numbers I told you to write down that day as variable y, if you weren't in class that day instead do X" Then you are measuring class attendance rather than subje…

They aren't penalized, they can still do the X alternative that relies only on understanding the material. This is merely a method for determining who to take a closer look at, not what that closer look will reveal.

Re: College students are learning hard lessons about anti-cheating software

#60

Post by ht_th »

ghaff wrote: post #47
ht_th wrote: post #31

Earlier quoted context omitted.

Sure. Have at most ten to twenty students per teacher. Then the teacher can spend a couple of hours per student to assess their understanding. Unfortunately, in education quantity seems to be prioritized over quality. When you have over 50 students, let alone over 200 hundred, good luck grading authentic open-ended projects or papers in just two weeks. As a result, most tests consists of multiple-choice questions and…

> Then the teacher can spend a couple of hours per student to assess their understanding. That sounds as if you're steering close to a whiteboarding interview which a lot of people will claim isn't great. Take home tests aren't perfect but I had plenty of them in school. And even when I didn't I don't think I had just about any engineering class that didn't assign problem sets that counted for a decent portion of you…

Taking home exams or assignments are great, but they need to be assessed as well. Here too, the number of students is a limiting factor. With too many students, the best a teacher can do is summative assessment and generate a reasonable grade for the student. Personally, I think formative assessment is far superior to summative assessment, but formative assessment takes a lot of effort from the teacher. Often more than is allotted to her for assessement (or the whole course!).

I was thinking more about a larger project or paper with quite a lot of freedom for the students that would take the teacher hours to get familiar with and give constructive formative feedback on. As part of that, an oral section to discuss the project and paper with the student would be great as well. In assessment, I would like the teacher and student to be partners rather than opponents.

If a student understands the material, together they should be able to have a conversation where the student can reflect on her understanding and learning to the extent where the teacher can determine that the student has passed the course. Of course, if there is an intense interaction between the student and teacher throughout the course, the whole idea of a final assessment becomes pretty meaningless.

I know, my ideas are a bit far-fetched and there are a lot of practicalities that are difficult to work out. But one can dream!

Post reply on HN