Live data from Hacker News

U-M finds students with alphabetically lower-ranked names receive lower grades

record.umich.edu

41–50 of 284 posts

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#41
I work in academia. When we grade exams, the order of the exams on the stack is the order in which they were collected in the room (people can sit wherever they like). For grading, we are usually 5 people in a single room, and everyone grades a specific exercise for consistency. The exams are getting shuffled heavily, with everyone just grabbing stacks, looking for exams where "their" exercise was not yet graded, and taking them out. So basically, the order in which we grade exams can be considered random.

However, I also grade weekly exercise sheets during the semester, and these are committed into a repository, where each student has a folder that... begins with the first letter of their first name. Everyone I have ever worked with acknowledges that you have to shuffle the order in which you grade these submissions each week, for fairness. Several effects come into play: (1) your are usually less tired at the beginning, (2) your mood gets better during the last 2 sheets because you know you are done soon, (3, and crucially) at the beginning, you have not yet seen all the common errors / developed a "feeling" for them, and you might thus miss them in early submissions, but spot them immediately in later submissions.

Another alphabetic effect: In elementary school, my name was on top of the list of students in my class. I remember that I often had to do some special job simply because I was the first name on this list (for example, carry a group ticket when we visited some museum, keep track of something, be the first at something where nobody wanted to be the first, with everyone watching, be the first to be graded in PE, again with everyone watching, etc.). As a fairly shy kid, this already annoyed me in first grade.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#42
post #15

As someone whose initials are Z and W, I tend to notice alpha sort a lot. Asking a friend whose initials are A and B about this, it's not something they ever noticed. I haven't noticed a grading/ranking difference, but far more frequently I'll hear that "oh, we ran out of item/time/etc. before we got to you", which has made me much more sensitive to issues of planning/organization.

Outside of school I can’t think of even a single instance of alphabetical sorting of my name (I have a middle letter). What situations are you in that this comes up a lot?

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#43
post #30

Earlier quoted context omitted.

I was a grader once. I guarantee if someone gives a good answer they'll get full marks even near the bottom of the stack. For BS answers I'll admit I got less generous as the hours went on. No one's getting hurt by this system if it's randomized. It's a matter of graders giving out partial credit for wrong answers which is discretionary. Rarely students are granted a small mercy. Seems OK.

> For BS answers I'll admit I got less generous as the hours went on. What do you think is the cause of this? Do you become more cynical (and less generous) because you’ve seen so many BS answers previously? Is it just that getting fatigued makes you less generous?

When I was a TA in grad school, I noticed the same. Early on I thought some BS answers were at least kind of funny, and I gave them the benefit of the doubt, maybe giving more attention to the parts that were correct. After I saw similar answers later on, the novelty wore off and I was probably less amused, so the inclination to be lenient disappeared. Sometimes I went back to previous decisions if I remembered them, to be fair, but I don't think I always remembered since the volume could be high (grading 80 exams in a row is TEDIOUS).

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#45
post #38
post #5

>One simple fix would be to make random order the default setting. Fixed in the sense that the bias will be random. Presumably students graded last will still receive lower grades.

It should average out over their career at the university - whereas if the alphabetical order is kept, then they would be systematically penalised.

It won’t average out perfectly. There will still be lucky and unlucky students.

Of course it’s better than a fixed order, and if it’s easy to switch then might as well. But we should keep thinking about how we can make it even better.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#46

Earlier quoted context omitted.

> Enforced randomization isn't going to fix the problem, it just evenly distributes the problem. Evenly distributing the problem does fix the problem. Proportionality is what matters. Grading being arbitrary is fine if everyone is graded equally.

Random order would still mean a few students in the class get unlucky and near the end the majority of the time. Although over the course of all classes it would tend to even out somewhat. It’s certainly better than fixed order.

[deleted]

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#47
post #33

I'm willing to take bets that in 15 years there will be a scandal about faked data by at least one of the researches in this paper. It smell just like every other interesting psychology result that at best is a fluke.

Unlikely. If you talk with anyone who's done grading, this will likely jive with our experience and make us data aware of the outcomes. Like anything, with grading you can get into a flow, and the more you process an assignment, the more answers you've seen and those can change how you grade future answers

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#48
post #2

> Wang noted that for a small group of graders (about 5%) that grade from Z to A, the grade gap flips as expected This is critical. Otherwise we could not discount some group (e.g. some ethnicity) disproportionately occupying one end of the alphabet or another. Super interesting and important finding. I hope this gets wide visibility and universities take a break from politicking to fix the problem - presumably throu…

Enforced randomization isn't going to fix the problem, it just evenly distributes the problem. Based on these results, it would mean that the graders are just getting tired/lazy/inattentive the further they get in their stack of papers to grade. That's the problem the needs to be fixed, not the order they get graded in. Enforced randomization is simply a short term alleviation so no student(s) get disproportionately…

In my experience, it's not tired/lazy/inattentive, but resignation. You normally have some expectation what students will be able to solve. Typically, these expectations are set too high. That's very common, not only for me, but for pretty much anyone I know. So over the time of grading, one adjusts down the expectations and gives partial credit earlier, for example.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#49

Earlier quoted context omitted.

Enforced randomization isn't going to fix the problem, it just evenly distributes the problem. Based on these results, it would mean that the graders are just getting tired/lazy/inattentive the further they get in their stack of papers to grade. That's the problem the needs to be fixed, not the order they get graded in. Enforced randomization is simply a short term alleviation so no student(s) get disproportionately…

> graders are just getting tired/lazy/inattentive the further they get in their stack of papers to grade. I will admit to this. Initially, my patience and tolerance for errors is significantly higher than towards the end of the grading. By the second hour grading, I am not only mentally exhausted my tolerance is significantly lower. I try to prevent this by creating very explicit grading rubric and I stick to it as m…

Clear rubrics are the thing where possible. They aren't everywhere though. I've been on conference committees and so many different factors come into play--including how late in the day it is. But, in that case, a bunch of people are rating and commenting and there's no strict order so it probably evens out to a reasonable degree.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#50

Earlier quoted context omitted.

Enforced randomization isn't going to fix the problem, it just evenly distributes the problem. Based on these results, it would mean that the graders are just getting tired/lazy/inattentive the further they get in their stack of papers to grade. That's the problem the needs to be fixed, not the order they get graded in. Enforced randomization is simply a short term alleviation so no student(s) get disproportionately…

For me I grade tests as follows. The stack is created as students turn in the test. I grade the first page in that order. The stack reverses for the second page. So on and so forth. I teach college math. I just cant imagine a system of grading done in alphabetical order.

Scanning and grading on a computer can alphabetize them.
Post reply on HN