U-M finds students with alphabetically lower-ranked names receive lower grades • Hacker News Board

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#91

Post by furyofantares » Sat, Apr 20, 2024, 3:06 PM UTC

zeroonetwothree wrote: post #70

furyofantares wrote: post #58

Earlier quoted context omitted.

Since the effect looks very small, it looks to me like it's only a problem because it adds up if it happens for every assignment for every course. I don't think it needs to average out perfectly; it looks to me like you'd have to be astronomically lucky/unlucky for it to matter if each assignment is in random order.

Some courses are only graded based on a small number of tests. I actually went to UM and a grade might be something like 30% midterm 60% final 10% homework (obviously different professors have different systems). In that case if you get unlucky just twice on the two tests you basically get the full penalty.

I'm not sure how much a +/- 0.3 (out of 100) deviation from average on a single course matters even if you end up dead first/last for both midterm and final in that example. I mean, it will matter sometimes. But it's (by far) not as big a deal as if it happens for all your courses.

Still, yes, you could flip the order from midterm to final instead of randomizing both and the effect goes to more like +/- 0.1 out of 100 for the luckiest and unluckiest.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#92

Post by xhkkffbf » Sat, Apr 20, 2024, 3:07 PM UTC

noodlesUK wrote: post #6

At my university, almost all of our marking was pseudonymised. We were assigned a random candidate number at the beginning of each year, and that is what went on our important papers/exams. The less important coursework often didn’t bother with this, and used our student numbers instead, but the general idea was the same. We didn’t put our names on any of our work other than our dissertation (and a few trivial assign…

I think the point is that some automated systems like Canvas may hide the names, but they're still presented in alphabetical order. Pseudonyms don't help if you don't shuffle them.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#93

Post by samatman » Sat, Apr 20, 2024, 3:07 PM UTC

A computer-based system like this is an opportunity to remove all personal details from an assignment while grading it, it baffles me that this isn't the default.

The database could tag every assignment with a UUID4, and present them for grading top-to-bottom in UUID lexical order, without exposing who is being graded in any way.

You can't fix fatigue bias, but this would distribute it randomly. It also removes the opportunity for favoritism and hostility, subconscious or otherwise, which is probably more important.

Once grading is completed, the assignments are reconnected with students. Give the profs a way to mark assignments with metadata, sometimes they need to talk to a student personally about something, this should be made easy.

Grades can't be immutable, professors need discretion in that, but it would leave an audit trail if professors maliciously modified grades (or the opposite). That should be uncommon to begin with, but both professors and students benefit from an audit trail here.

A system like this should be used whenever it's practical, and always for high-stakes tests like midterms and finals. Not making a case against oral exams here, just that when it's possible to blind the grading process, it should be.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#94

Post by stikit » Sat, Apr 20, 2024, 3:08 PM UTC

A .3 point difference isn’t going to make a real difference to anyone’s life and is likely a wash when other yet undiscovered biases are in the mix. Unfairness and bias is a critical factor in driving people to extraordinary achievements.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#95

Post by faitswulff » Sat, Apr 20, 2024, 3:08 PM UTC

I wonder if these biases are replicable in LLMs.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#96

Post by somenameforme » Sat, Apr 20, 2024, 3:09 PM UTC

verdverm wrote: post #47

llm_trw wrote: post #33

I'm willing to take bets that in 15 years there will be a scandal about faked data by at least one of the researches in this paper. It smell just like every other interesting psychology result that at best is a fluke.

Unlikely. If you talk with anyone who's done grading, this will likely jive with our experience and make us data aware of the outcomes. Like anything, with grading you can get into a flow, and the more you process an assignment, the more answers you've seen and those can change how you grade future answers

Not really taking a position on this one way or the other, but I would say that "this jives with my experience" is near to being a prerequisite for junk science. Somebody saying something controversial is going to be challenged -- confirming biases is precisely how you peddle junk.

For instance the Journal of Personality and Social Psychology [1] is a terrible journal, with a replication success rate in the 20% range. Yet it's ironically well regarded. Both can probably be explained by the exact same phenomena - go read their articles and reads like a stream of bias confirmations for those of a certain ideological orientation -- the same orientation that's clearly widely shared amongst social science researchers.

[1] - https://psycnet.apa.org/PsycARTICLES/journal/psp/126/2

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#97

Post by mistrial9 » Sat, Apr 20, 2024, 3:09 PM UTC

current curricular trends in California include "algebra removed from 8th grade as unfair" (or more extreme rhetoric given) and this week "equity grading for K-8" where there is no D or F given in any subject. These real-life changes combined with something so arbitrary as this one as "news" really give an impression of a collapse of some kind in public education discourse.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#98

Post by samatman » Sat, Apr 20, 2024, 3:10 PM UTC

xmddmx wrote: post #87

Is anyone confused by "lower-ranked names"? To me this means A, B, C, but the article says "Wang said students whose surnames start with A, B, C, D or E received a 0.3-point higher grade out of 100 possible points than compared with when they were graded randomly." So I guess "alphabetically lower ranked" means the last letters of the alphabet, not first? Confused.

This is an important observation!

The programmer's perspective and the user's perspective aren't always the same, and both need consideration. A user is going to see a list: it starts at the top, and it ends at the bottom. The first fields are higher, the later fields are lower.

Of course, if this is a sorted list, the first field will be the "lowest" value, for whatever comparison is used to sort it.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#99

Post by ghaff » Sat, Apr 20, 2024, 3:13 PM UTC

1-6 wrote: post #52

klysm wrote: post #24

Job interviews have similar effects

Order matters a lot but recruiters typically present the highest flyers first and the lower candidates last.

In my experience, it varies. I've been on interview panels where we just weren't feeling it for a number of candidates and basically told the recruiter to try harder and eventually hit someone who we were "That's who we want. Find a way to make it happen."

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#100

Post by dunham » Sat, Apr 20, 2024, 3:15 PM UTC

throwaway35777 wrote: post #27

buggy6257 wrote: post #4

Earlier quoted context omitted.

Enforced randomization isn't going to fix the problem, it just evenly distributes the problem. Based on these results, it would mean that the graders are just getting tired/lazy/inattentive the further they get in their stack of papers to grade. That's the problem the needs to be fixed, not the order they get graded in. Enforced randomization is simply a short term alleviation so no student(s) get disproportionately…

I was a grader once. I guarantee if someone gives a good answer they'll get full marks even near the bottom of the stack. For BS answers I'll admit I got less generous as the hours went on. No one's getting hurt by this system if it's randomized. It's a matter of graders giving out partial credit for wrong answers which is discretionary. Rarely students are granted a small mercy. Seems OK.

I was one of many TAs for a large math class in college (pre-calc - think high school math for college students). For uniformity, the prof had the partial credit down to a science - specifying points for getting certain aspects of the problem. For the finals, a few TAs would be assigned to a given page, for uniformity.

The fascinating thing was that the distribution of grades was about the same every year.

And I had a math prof for analysis who would give negative points for BS answers. You could say “I need X but don’t know how to prove it” in the middle of a proof, but if you made up something that was incorrect, you’d get negative points.