Live data from Hacker News

Do AI detectors work? Students face false cheating accusations

bloomberg.com

801–810 of 1001 posts

Re: Do AI detectors work? Students face false cheating accusations

#801
post #792

Earlier quoted context omitted.

What PhD program doesn't require a public defense? I'm currently a PhD candidate, and our program includes separate written and oral qualifying exams during the first year or two, and a public defense of the dissertation at the end. I thought some minor variation of this was nearly universal. It's also my observation, by the way, that the public dissertation defense (and even the written dissertation itself) is less…

Imagine Alan Turing's defense being a summary of 3 papers. The actual issue is that advanced education is increasingly not about doing fundamental scholarship but a pipeline for (re)producing a clerisy-intellectual class. There are a lot of leftist academics who point out this sea change in academia over the last century, see for example Norm Finkelstein's remarks on this but there are others who talk about this.

Oh yeah, there's a whole different discussion to be had (and HN does have it often), about the problems with peer reviewed publications and citations being the end-all for graduate students and professors.

My particular school and department is interesting because it doesn't have any hard requirement for publications, and it aims to have students finish a PhD in about three years of full-time work (assuming one enters the program with a relevant master's degree already in-hand). There has been some tension between the younger assistant professors (who are still fighting for tenure) and the older full professors (who got tenure in, say, the 1990s). In practice, the assistant professors expect to see their students publish (with the professors as co-authors, of course) and would strongly prefer to see a dissertation comprised of three papers stapled together, regardless of the what the school and department officially says. The full professors, on the other hand, seem to prefer something more like a monograph that is of "publishable" quality, maybe to be submitted somewhere after graduation. They argue that the assistant professors should be able to judge quality work for themselves instead of outsourcing it to anonymous reviewers. Clearly, there are different incentives at play.

Re: Do AI detectors work? Students face false cheating accusations

#802

I’ve been teaching in higher education for 30 years and am soon retiring. I teach math. In every math course there is massive amounts of cheating on everything that is graded that is not proctored in a classroom setting. Locking down browsers and whatnot does not prevent cheating. The only solution is to require face-to-face proctored exams and not allow students to use technology of any kind while taking the test. B…

Honestly, the problem is not the cheating, per se.

The problem is the lack of learning the material. You don't, IMO, directly care how they produced the answer, you care about it only as a proxy for them learning the material well enough to solve the problem.

And making people do them in person with no technology is unrealistic - not because it can't be done, but because at that point, it's not a reflection of how you'd use it outside of that classroom, and people are going to call it pointless, and IMO they'd be right. You would be correct that anyone who met that bar would have likely learned the material, but you'd also have excluded people who would have met the bar of "can use the material to the degree of familiarity needed going forward".

I think a reasonable compromise would be to let students collaborate on the exams in the classroom, without external access - while I realize some people learn better on their own in some subjects, as long as everyone contributes some portion of the work, and they go back and forth on agreeing what the right answers are, then you're going to make forward progress, even if that ruins the exam process as anything other than a class-wide metric. You could subdivide it, but then that gets riskier as there's a higher chance that the subset of people doesn't know enough to make progress. Maybe a hint system for groups, since the goal here is learning, not just grading their knowledge going in?

Not that there's not some need for metrics, but in terms of trying to check in on where students are every so often, I think you need to leverage how people often end up learning things "in the wild" - from a combination of wild searching and talking to other people, and then feedback on whether they decided you could build an airplane out of applesauce or something closer to accurate.

Re: Do AI detectors work? Students face false cheating accusations

#803
post #448
post #265

Earlier quoted context omitted.

That depends on the questions. There are also open book exams. A viva is a type of exam so I don’t see they are incompatible with assessing research

Not every class is STEM. Are you writing a 4000 word research paper sitting in class?

Some of my experience of exams comes from a history degree where around eighty to ninety percent of the overall grade came from final exams. I can only speak of my experience but I don’t think this is atypical depending on educational system.

One of the reasons I mentioned the viva was an example of how we can decouple production of some work from an assessment of quality and some reason to believe that some candidate is capable of the work without assistance.

It would be unreasonable to spend five or so years working under examination conditions. But that doesn’t mean we can’t subsequently examine a candidate to determine likely authorship amongst other things.

Re: Do AI detectors work? Students face false cheating accusations

#804

Earlier quoted context omitted.

So if an automated decision happens, and the reviewer looks for a second at it, and says, good enough, that will be OK according to GDPR. Don't see what GDPR solves here.

Well I guess the theory is that you could go to court, and the court would be reasonable and say "this 1 second look does not fulfill the requirement, you need to actually use human judgement and see what was going on there". Lots of discussions regarding FAANG malicious compliance have shown this is how high courts work in EU. When there is political will. But if you're a nobody, and can't afford to go to court agai…

I don't see that even people with money and power can do anything here. It is like VAR. When has it ever happened that the referee goes to the screen, and does not follow the VAR recommendation? Never. That is how automated decision making will work as well, across the board.

Re: Do AI detectors work? Students face false cheating accusations

#805

Earlier quoted context omitted.

Serious question from someone who is regularly tasked with hiring Juniors. What IS a good assessment for entry-level/right out of college positions? -> GPA can be gamed, as laid out. -> Take Home assessments can mostly be gamed, I want to assess how you think, now which tools you use. -> Personality tests favor the outgoing/extroverts -> On-location tests/leet code are a crapshoot. What should be best practice here?…

I think the best test for a Junior is to ask them to submit some of their OSS or personal fun projects they've worked on. From my perspective, especially with Juniors who aren't expected to be extremely knowledgeable, displaying a sense of curiosity and a willingness to learn is much more important. If, hypothetically, there's two candidates, one who is more knowledgeable but has no personal projects versus someone w…

Most engineers, including good ones, that I've interviewed have no interesting GitHub contributions. GitHub is also game-able. Bootcamps, in particular, push their graduates to build an interesting GitHub portfolio.

I've found that talking through projects is a weak indicator of competence. It's much easier to memorize talking points than to produce working code.

Re: Do AI detectors work? Students face false cheating accusations

#806
post #732

Earlier quoted context omitted.

I did a "hard" degree and saw classmates who worked half as hard sail by me, because they cheated. Groups that share answer banks, in-class quizzes with answers shared (when they were not supposed to be), group projects that used last year's stuff. All of it, all the way through final exams, which people had answer keys to. I had a few classmates that were formally investigated for cheating by the university; their p…

You're supposed to work as hard as you can, then cheat for the grade.

No, you're really not supposed to cheat.

Re: Do AI detectors work? Students face false cheating accusations

#807
post #611

I’ve been teaching in higher education for 30 years and am soon retiring. I teach math. In every math course there is massive amounts of cheating on everything that is graded that is not proctored in a classroom setting. Locking down browsers and whatnot does not prevent cheating. The only solution is to require face-to-face proctored exams and not allow students to use technology of any kind while taking the test. B…

My personal take, we’ve made the cost of failure to high and cheating too easy. As a student, the only thing the next institution will see is GPA, school, major. Roughly in that order. If the cost of not getting an A is exclusion from future opportunities- then students will reject exclusion by taking easier classes or cheating. As someone who studied physics and came out with a 2.7 GPA due to studying what I wanted…

I just want to second this (also did an undergrad in physics funny enough). I specifically sought out the harder professors in my undergrad and for the most part I'm happy I did it, but it's also a good thing that I'm not very motivated by money or prestige because I saw many of my colleagues who had gotten into better schools or jobs (even just the return calls on applications) who chose the easier routes or cheated. They are without a doubt wealthier. What mattered the most was the line items on their resumes and networking, but there is feedback in this so one begets the other. Fwiw, I had a 3.3.

So it then becomes hard for me to make suggestions to juniors. It isn't difficult to sniff out those like you or me who are motivated by the rabbit holes themselves, nor difficult to tell those who are entirely driven by social pressures (money, prestige, family, etc), but what about those on the edge? I think it's the morally best option to encourage learning for learning but it's naive to also not recognize that their peers who will cheat will be rewarded for that effort. It's clear that we do not optimize for the right things and we've fallen victim to Goodhart's Law, but I just hope we can recognize it because those systems are self reinforcing and the longer we work in them the harder they are to escape. Especially because there are many bright students who's major flaw is simply a lack of opportunity. For me? I'm just happy if I can be left to do my research, read papers and books, and have sufficient resources -- which is much more modest than many of my peers (ML). But it'd be naive to not recognize the costs and I'm a big believer in recognizing incentive structures and systematic issues. Unfortunately these are hard to resolve because they're caused by small choices by all of us collectively, but fortunately that too means they can be resolved by small choices each of us make.

Re: Do AI detectors work? Students face false cheating accusations

#808

Earlier quoted context omitted.

Serious question from someone who is regularly tasked with hiring Juniors. What IS a good assessment for entry-level/right out of college positions? -> GPA can be gamed, as laid out. -> Take Home assessments can mostly be gamed, I want to assess how you think, now which tools you use. -> Personality tests favor the outgoing/extroverts -> On-location tests/leet code are a crapshoot. What should be best practice here?…

It's subtle, but people who are self driven and learning for the sake of learning will talk differently. They tend to include more nuance and detail, addressing the subtle things. To be able to see those things requires internalization of what's learned, not just memorization. If you get good at it, you can do pretty well at recognizing these people even when they're in a different subject domain. Remember, outside C…

> I'm also very sympathetic to the jitters stuff, because I get it a lot and make dumb mistakes when in the spotlight lol.

As an interviewer, I spot this and try to get them to ease up. I will talk about myself for a bit, about the work I do. I'm trying to get them to realize they are not in the spotlight, but whether we would be a good fit together; and thus both of us want them to work there.

BUT, my interviews tend to be about us solving a problem together, very rarely about actual code. For example, we might walk through how we would implement an email inbox system. We may discuss some of the finer details, if they come up, but generally, I'm interested in how they might design something they've basically used every day. How would we do search, what the database schema would look like, drafts, and so on.

I won't nudge them (to keep my biases in check), but I will help them down the path they choose, even if I don't like it. I'm not testing for the chosen path, but what "gotchas" they know and how they think though them. If you are a programmer, it shows. If you are an excellent programmer, it shows. If you are not a programmer, you won't make it 10 minutes.

Re: Do AI detectors work? Students face false cheating accusations

#809

Earlier quoted context omitted.

When I studied physics at Exeter University they still used the tutorial system and finals. Tutorials were held fortnightly; the tutorial groups were typically three or four students. There was no obligation to turn up to lectures or even tutorials. You just had to pass the end of year exams to be allowed to continue to the final. The class of degree that was awarded depended on the open note final exam and the repor…

Fellow UK person - the style of exam that you describe is pretty hard to cheat unless you can find another person to go in your place. I think various institutions have tried digital invigilation but have had little success (and I think this is just a bad idea anyway). However, you also mentioned a final project. You’d be shocked how much commissioning exists where people have their projects produced for them. I’m no…

My final year project was a 120 page report of measurements of electron spin resonance together with the design of the experimental apparatus. I had to defend the design, conclusions (which I have long forgotten, it was in 1977), and justify the methods and calculation all orally to two academics.

I doubt that anyone could have produced a plausible report without actually doing the work. And to defend it one would have to understand the underlying physics and the work that was done. Plus I think my supervisor and the other two students who worked with me on the project would have remarked on my absence from the laboratory if I had simply bought the paper!

You can still have long term projects and dissertations so long as the degree is awarded for the defence of the dissertation rather than the dissertation itself; that is the student must demonstrate in a viva that they understand everything in the dissertation rather than merely regurgitate it.

Re: Do AI detectors work? Students face false cheating accusations

#810
post #611

I’ve been teaching in higher education for 30 years and am soon retiring. I teach math. In every math course there is massive amounts of cheating on everything that is graded that is not proctored in a classroom setting. Locking down browsers and whatnot does not prevent cheating. The only solution is to require face-to-face proctored exams and not allow students to use technology of any kind while taking the test. B…

My personal take, we’ve made the cost of failure to high and cheating too easy. As a student, the only thing the next institution will see is GPA, school, major. Roughly in that order. If the cost of not getting an A is exclusion from future opportunities- then students will reject exclusion by taking easier classes or cheating. As someone who studied physics and came out with a 2.7 GPA due to studying what I wanted…

One of the smartest people I know did 4 degrees in 4.5 years: undergrads in physics, chem, biochem, and math. He graduated with like a 3.2 gpa, low because he took 18-22 credits of hard classes every single semester, and couldn't get into med school. They made him take some stupid biochem masters, at which he excelled, particularly with a reduced course load. He then easily got admitted to med school.

If you don't want people to prioritize grades over everything else...

Post reply on HN