Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

U-M finds students with alphabetically lower-ranked names receive lower grades

record.umich.edu

Post reply on HN
Print view View original

61–70 of 284 posts

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#61

Post by whiterknight »

BugsJustFindMe wrote: post #26
buggy6257 wrote: post #4

Earlier quoted context omitted.

Enforced randomization isn't going to fix the problem, it just evenly distributes the problem. Based on these results, it would mean that the graders are just getting tired/lazy/inattentive the further they get in their stack of papers to grade. That's the problem the needs to be fixed, not the order they get graded in. Enforced randomization is simply a short term alleviation so no student(s) get disproportionately…

> Enforced randomization isn't going to fix the problem, it just evenly distributes the problem. Evenly distributing the problem does fix the problem. Proportionality is what matters. Grading being arbitrary is fine if everyone is graded equally.

The mistake is assuming grades are an objective measurement, and not gamification to try to help you learn.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#63

Post by zeroonetwothree »

verdverm wrote: post #47
llm_trw wrote: post #33

I'm willing to take bets that in 15 years there will be a scandal about faked data by at least one of the researches in this paper. It smell just like every other interesting psychology result that at best is a fluke.

Unlikely. If you talk with anyone who's done grading, this will likely jive with our experience and make us data aware of the outcomes. Like anything, with grading you can get into a flow, and the more you process an assignment, the more answers you've seen and those can change how you grade future answers

I really doubt you can notice a 0.6% discrepancy anecdotally. They only detected it in the study because of the massive amount of data they used.

Classic confirmation bias.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#64

Post by [deleted] »

davrosthedalek wrote: post #48
buggy6257 wrote: post #4

Earlier quoted context omitted.

Enforced randomization isn't going to fix the problem, it just evenly distributes the problem. Based on these results, it would mean that the graders are just getting tired/lazy/inattentive the further they get in their stack of papers to grade. That's the problem the needs to be fixed, not the order they get graded in. Enforced randomization is simply a short term alleviation so no student(s) get disproportionately…

In my experience, it's not tired/lazy/inattentive, but resignation. You normally have some expectation what students will be able to solve. Typically, these expectations are set too high. That's very common, not only for me, but for pretty much anyone I know. So over the time of grading, one adjusts down the expectations and gives partial credit earlier, for example.

[deleted]

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#65

Post by zeroonetwothree »

llm_trw wrote: post #33

I'm willing to take bets that in 15 years there will be a scandal about faked data by at least one of the researches in this paper. It smell just like every other interesting psychology result that at best is a fluke.

I think it’s maybe less likely since this is looking at actual grades and not some kind of survey or experiment. But certainly it’s always a concern in social sciences until we get reproduction.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#66

Post by kurthr »

skhunted wrote: post #29
buggy6257 wrote: post #4

Earlier quoted context omitted.

Enforced randomization isn't going to fix the problem, it just evenly distributes the problem. Based on these results, it would mean that the graders are just getting tired/lazy/inattentive the further they get in their stack of papers to grade. That's the problem the needs to be fixed, not the order they get graded in. Enforced randomization is simply a short term alleviation so no student(s) get disproportionately…

For me I grade tests as follows. The stack is created as students turn in the test. I grade the first page in that order. The stack reverses for the second page. So on and so forth. I teach college math. I just cant imagine a system of grading done in alphabetical order.

I also came here to say this. My only guess is that the alphabetization (by the "learning management system") to make filling the grades into a table "easier" for the computer or for the person handing out the results? Why is it "easier" if the system doesn't have to order them at all, or it could do so by student number (same issue as alphabetical order) or something random, which is the other (non default) option for the "learning management system".

I feel like only the most obsessive compulsive humans would have this issue (without computer "help"), as the last thing I wanted to do as a TA was to add another step of ordering all the papers before grading them. I also always reviewed the first few papers I graded after grading the rest to make sure I was being fair, because it was obvious to me that until I saw a representative distribution of answers I couldn't do fair grading/marking.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#67

Post by cyanydeez »

buggy6257 wrote: post #4
hilux wrote: post #2

> Wang noted that for a small group of graders (about 5%) that grade from Z to A, the grade gap flips as expected This is critical. Otherwise we could not discount some group (e.g. some ethnicity) disproportionately occupying one end of the alphabet or another. Super interesting and important finding. I hope this gets wide visibility and universities take a break from politicking to fix the problem - presumably throu…

Enforced randomization isn't going to fix the problem, it just evenly distributes the problem. Based on these results, it would mean that the graders are just getting tired/lazy/inattentive the further they get in their stack of papers to grade. That's the problem the needs to be fixed, not the order they get graded in. Enforced randomization is simply a short term alleviation so no student(s) get disproportionately…

Unfortunately, it's gonna be AI to the "rescue" and the problem is obfuscated.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#68

Post by COGlory »

Multiple factors at play here.

1) Rubrics are often defined, but the application of the rubric is by a human. Application will shift as the grader gets a sense of the classes understanding.

2) As you get fatigued while grading, you'll make mistakes, and be less tolerant of others. Especially if you're an overworked adjunct or graduate student.

3) There are probably a lot more last names early in the alphabet so weighting is important.

My policy on this when I was a grad student was to publish the rubric, and ask all students to check their grades too.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#69

Post by BugsJustFindMe »

whiterknight wrote: post #61
BugsJustFindMe wrote: post #26

Earlier quoted context omitted.

> Enforced randomization isn't going to fix the problem, it just evenly distributes the problem. Evenly distributing the problem does fix the problem. Proportionality is what matters. Grading being arbitrary is fine if everyone is graded equally.

The mistake is assuming grades are an objective measurement, and not gamification to try to help you learn.

It's a common mistake. So common, in fact, that it has real practical impact on students at the edge who might not otherwise have failed or passed.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#70

Post by zeroonetwothree »

furyofantares wrote: post #58
zeroonetwothree wrote: post #45

Earlier quoted context omitted.

It won’t average out perfectly. There will still be lucky and unlucky students. Of course it’s better than a fixed order, and if it’s easy to switch then might as well. But we should keep thinking about how we can make it even better.

Since the effect looks very small, it looks to me like it's only a problem because it adds up if it happens for every assignment for every course. I don't think it needs to average out perfectly; it looks to me like you'd have to be astronomically lucky/unlucky for it to matter if each assignment is in random order.

Some courses are only graded based on a small number of tests. I actually went to UM and a grade might be something like 30% midterm 60% final 10% homework (obviously different professors have different systems). In that case if you get unlucky just twice on the two tests you basically get the full penalty.
Post reply on HN