Do AI detectors work? Students face false cheating accusations • Hacker News Board

Re: Do AI detectors work? Students face false cheating accusations

#751

Post by medmunds » Mon, Oct 21, 2024, 6:05 PM UTC

pj_mukh wrote: post #669

lumost wrote: post #611

Earlier quoted context omitted.

My personal take, we’ve made the cost of failure to high and cheating too easy. As a student, the only thing the next institution will see is GPA, school, major. Roughly in that order. If the cost of not getting an A is exclusion from future opportunities- then students will reject exclusion by taking easier classes or cheating. As someone who studied physics and came out with a 2.7 GPA due to studying what I wanted…

Serious question from someone who is regularly tasked with hiring Juniors. What IS a good assessment for entry-level/right out of college positions? -> GPA can be gamed, as laid out. -> Take Home assessments can mostly be gamed, I want to assess how you think, now which tools you use. -> Personality tests favor the outgoing/extroverts -> On-location tests/leet code are a crapshoot. What should be best practice here?…

It of course depends on what you’re hiring for, what qualities you value, and the scale you’re working at. But:

> I want to assess how you think, not which tools you use

suggests you have a more nuanced approach and aren’t just aiming for large numbers of drones.

What worked well for me (in a couple of smaller companies/teams) was:

- Talk to the candidates about their experiences in a project-oriented course where they had to work in a team. (Most CS programs have at least one of these. Get the name of that course ahead of time and just ask about it.) You want to find out if they can work in a team, divide up work and achieve interim goals, finish a project, deal with conflicts, handle setbacks and learn from mistakes, etc.

- Similarly, find out the names of some of the harder elective courses, and ask about their experiences in these. This gets at what they find interesting, how they think, and can help filter out GPA gamers.

- Talk to them about their experiences in whatever jobs, internships, volunteer work, or extracurricular activities they engaged in while at school. It doesn’t have to be directly related to your field—-you’re screening for work ethic and initiative.

Admittedly it’s been a while, but we used this approach for both on-campus recruiting and remote phone screens, and got pretty good at hitting these topics in a 15-20 minute conversation. We’d have one or two people screen maybe 30-50 candidates each recruiting season, identify 5-10 for on-site interviews with a larger team, and end up hiring about half of those.

This sort of bespoke screening does take some work on your part, and can be tough to scale. But we found it consistently identified solid candidates and led to outstanding hires.

Re: Do AI detectors work? Students face false cheating accusations

#752

Post by godelski » Mon, Oct 21, 2024, 6:06 PM UTC

pj_mukh wrote: post #669

lumost wrote: post #611

Earlier quoted context omitted.

My personal take, we’ve made the cost of failure to high and cheating too easy. As a student, the only thing the next institution will see is GPA, school, major. Roughly in that order. If the cost of not getting an A is exclusion from future opportunities- then students will reject exclusion by taking easier classes or cheating. As someone who studied physics and came out with a 2.7 GPA due to studying what I wanted…

Serious question from someone who is regularly tasked with hiring Juniors. What IS a good assessment for entry-level/right out of college positions? -> GPA can be gamed, as laid out. -> Take Home assessments can mostly be gamed, I want to assess how you think, now which tools you use. -> Personality tests favor the outgoing/extroverts -> On-location tests/leet code are a crapshoot. What should be best practice here?…

It's subtle, but people who are self driven and learning for the sake of learning will talk differently. They tend to include more nuance and detail, addressing the subtle things. To be able to see those things requires internalization of what's learned, not just memorization. If you get good at it, you can do pretty well at recognizing these people even when they're in a different subject domain.

Remember, outside CS no one else does whiteboard interviews or takehome tests. It's generally a few conversations and that's it. It's because experts been sniff out other experts in their domain fairly quickly. It's about *how* they think, not what they know.

I'll give you an example of something subtle but is a frequent annoyance for me and I'm sure many others. You're on a webpage that asks for your country. Easy, just put in a drop-down, right? But what's much much better it's to use the localization information of the browser to place a copy of that country at the top of the list (a copy, not move). Sure, it saves us just scrolling to the bottom, but my partner is Korean and she never knows if she's looking for K(orea), S(outh Korea), or R(epublic of Korea). This happens for a surprising number of countries. Each individual use might just save a second or two of time, but remember you also need to multiply that by the number of times people interact with that page, so it can be millions of seconds. It'll also just leave users far less frustrated.

I'm also very sympathetic to the jitters stuff, because I get it a lot and make dumb mistakes when in the spotlight lol. But you can often find these things in other work they've done if they include their GitHub. Even something small like a dotfiles repo. And if the interview is more about validation their experience, the attention to detail and deeper knowledge will still show up in discussions especially if you get them to talk about something they're passionate about.

I'd also say that GPA and school names are very noisy (incidentally that often means internships too, since these strongly correlate). I know plenty of people from top 3 schools who do not know very basic things but have done rounds at top companies and can do leet code. But they're like GPT or people who complain about math word problems, they won't generalize and recognize these things in the wild. Overfit and studies to the test (this is a subtle thing you can use while interviewing too)

Re: Do AI detectors work? Students face false cheating accusations

#753

Post by simondw » Mon, Oct 21, 2024, 6:07 PM UTC

_fat_santa wrote: post #576

skhunted wrote: post #491

I’ve been teaching in higher education for 30 years and am soon retiring. I teach math. In every math course there is massive amounts of cheating on everything that is graded that is not proctored in a classroom setting. Locking down browsers and whatnot does not prevent cheating. The only solution is to require face-to-face proctored exams and not allow students to use technology of any kind while taking the test. B…

I remember taking a math class in college and the professor had a very unique way of dealing with cheating. He let us use our books, notes, and "any calculator capability" from our TI-84's. His rationale is that students will try to use these tricks anyways so just let them and then update the test to be "immune" from these advantages. Before every test he mentioned that we could use all those tools but always said "…

That makes sense when tools are as dumb as static notes and TI-84s.

But in the (hypothetical) limit where AI tools outperform all humans, what does this updated test look like? Are we even testing the humans at that point?

Re: Do AI detectors work? Students face false cheating accusations

#754

Post by clarebear123 » Mon, Oct 21, 2024, 6:09 PM UTC

withinboredom wrote: post #747

Syonyk wrote: post #709

Earlier quoted context omitted.

It's hard, and interviewing is better suited to answering "nope, not you!" questions than "yes, you'll be a good fit." Onsite interviews with a range of approaches seem to be the best I've found over the years. As much as it pains me, things like fizzbuzz are still useful, because people still lie about their ability to program in languages. If you claim to know C very well and can't knock that out in 5 minutes, and…

> If you claim to know C very well and can't knock that out in 5 minutes, and it takes you 45 minutes of prompting, well, you don't know C usefully. I recently had an interview and a "skill test" in C. It was proctored by the interviewer in-person. I had so many questions about the questions. It was like, what is the output of "some code" and while obvious, there were some questions where specific CPU architecture ma…

What architectures would it not be 04 on?

Re: Do AI detectors work? Students face false cheating accusations

#755

Post by nonameiguess » Mon, Oct 21, 2024, 6:09 PM UTC

delusional wrote: post #199

lwhi wrote: post #178

It is no longer effective to solely use a written essay to measure how deeply a student comprehends a subject. AI is here to stay; new methods should be used to assess student performance. I remember being told at school, that we weren't allowed to use calculators in exams. The line provided by teachers was that we could never rely on having a calculator when we need it most—obviously there's irony associated with ha…

> I remember being told at school, that we weren't allowed to use calculators in exams I remember being told the same thing, but I happen to believe that it was a fantastic policy, with a lackluster explanation. The idea that you wouldn't have a calculator was obviously silly, even at the time, but underlying observation that relying on the calculator would rob you of the mental exercise the whole ordeal was supposed…

How old are you? And, for that matter, how old is the person you're responding to? In 1998, at least up to a TI-81 was allowed on the AP Calculus Exam (possibly higher than that, but you couldn't use anything that was programmable). I have to think it's been a very long time since no calculators at all were allowed for math exams unless you're talking arithmetic exams in elementary school where the entire point is to test how well you've memorized times tables or can perform manual long division.

Re: Do AI detectors work? Students face false cheating accusations

#756

Post by methodical » Mon, Oct 21, 2024, 6:09 PM UTC

pj_mukh wrote: post #669

lumost wrote: post #611

Earlier quoted context omitted.

My personal take, we’ve made the cost of failure to high and cheating too easy. As a student, the only thing the next institution will see is GPA, school, major. Roughly in that order. If the cost of not getting an A is exclusion from future opportunities- then students will reject exclusion by taking easier classes or cheating. As someone who studied physics and came out with a 2.7 GPA due to studying what I wanted…

Serious question from someone who is regularly tasked with hiring Juniors. What IS a good assessment for entry-level/right out of college positions? -> GPA can be gamed, as laid out. -> Take Home assessments can mostly be gamed, I want to assess how you think, now which tools you use. -> Personality tests favor the outgoing/extroverts -> On-location tests/leet code are a crapshoot. What should be best practice here?…

I think the best test for a Junior is to ask them to submit some of their OSS or personal fun projects they've worked on. From my perspective, especially with Juniors who aren't expected to be extremely knowledgeable, displaying a sense of curiosity and a willingness to learn is much more important.

If, hypothetically, there's two candidates, one who is more knowledgeable but has no personal projects versus someone who has less knowledge but has worked on different side projects in various languages/domains, I'm always going to pick the latter candidate since they clearly have a passion, and that passion will drive them to pick up the knowledge more than someone who's just doing it for a paycheck and could care less about expanding their own knowledge.

To go one step forward, you can ask them to go into detail about their side project, interesting problems they faced, how they overcame them, etc. Even introverts who are generally worse at small talk are on a much more balanced playing field when talking about something they're passionate about.

Re: Do AI detectors work? Students face false cheating accusations

#757

Post by BobaFloutist » Mon, Oct 21, 2024, 6:10 PM UTC

dmd wrote: post #438

ninalanyon wrote: post #365

Earlier quoted context omitted.

When I studied physics at Exeter University they still used the tutorial system and finals. Tutorials were held fortnightly; the tutorial groups were typically three or four students. There was no obligation to turn up to lectures or even tutorials. You just had to pass the end of year exams to be allowed to continue to the final. The class of degree that was awarded depended on the open note final exam and the repor…

What about those of us who can explain our ideas and thinking clearly and in great detail in writing but would struggle to even prove we've heard of the topic orally?

Even with extensive notes and prep-time in a one-on-one?

Can you communicate it in real-time through writing? Maybe that's an accommodation that could be done?

Re: Do AI detectors work? Students face false cheating accusations

#758

Post by curiouscavalier » Mon, Oct 21, 2024, 6:11 PM UTC

_ea1k wrote: post #624

lumost wrote: post #611

Earlier quoted context omitted.

My personal take, we’ve made the cost of failure to high and cheating too easy. As a student, the only thing the next institution will see is GPA, school, major. Roughly in that order. If the cost of not getting an A is exclusion from future opportunities- then students will reject exclusion by taking easier classes or cheating. As someone who studied physics and came out with a 2.7 GPA due to studying what I wanted…

This really can't be emphasized enough. Universities and the initial hiring process really optimize for a score and not for learning. Those could be, and sometimes are, correlated, but it isn't necessarily the case. Really focusing on stretching yourself necessarily means lower grades. Why is that penalized? TBH, in software engineering a lot of people with lower grades tutor the ones with 4.0 averages. The skillsets…

And it penalizes in many ways. Focusing too much on grades can be detrimental in graduate studies, despite graduate admissions focusing on GPA and test scores. I remember seeing 4.0 undergrads really struggle with research in grad school, sometimes to the point of dropping out. Certainly not always the case, but for the ones that did I think it speaks to your point about different skillsets.

Maybe worse was seeing the undergrads who passed on research opportunities out of fear it would distract them from keeping a high GPA.

Re: Do AI detectors work? Students face false cheating accusations

#759

Post by commandlinefan » Mon, Oct 21, 2024, 6:14 PM UTC

Syonyk wrote: post #709

pj_mukh wrote: post #669

Earlier quoted context omitted.

Serious question from someone who is regularly tasked with hiring Juniors. What IS a good assessment for entry-level/right out of college positions? -> GPA can be gamed, as laid out. -> Take Home assessments can mostly be gamed, I want to assess how you think, now which tools you use. -> Personality tests favor the outgoing/extroverts -> On-location tests/leet code are a crapshoot. What should be best practice here?…

It's hard, and interviewing is better suited to answering "nope, not you!" questions than "yes, you'll be a good fit." Onsite interviews with a range of approaches seem to be the best I've found over the years. As much as it pains me, things like fizzbuzz are still useful, because people still lie about their ability to program in languages. If you claim to know C very well and can't knock that out in 5 minutes, and…

> things like fizzbuzz are still useful

I think you're right here but, to play devil's advocate... isn't there some survivorship bias going on here? I assume you've never tested the negative hypothesis and gone ahead and hired somebody who couldn't program fizzbuzz to validate your assumption.

Re: Do AI detectors work? Students face false cheating accusations

#760

Post by ta1243 » Mon, Oct 21, 2024, 6:15 PM UTC

TeaBrain wrote: post #749

ta1243 wrote: post #612

Earlier quoted context omitted.

Sorry but nobody uses "cursive" writing. People barely write once they leave education - they type. When they do write it's legible separate characters or it's unreadable scrawl.

My brother (mid-20s) still writes in cursive. He does so beautifully, completely legibly, and rapidly, at a faster rate than he, or the average person, can write non-cursive.

What a useful skill for the 160 words a year the average person writes /s