U-M finds students with alphabetically lower-ranked names receive lower grades • Hacker News Board

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#31

Post by dgacmu » Sat, Apr 20, 2024, 2:35 PM UTC

jimmar wrote: post #13

Order effects are real. I'm a prof. I notice that the longer I grade, the less motivated I am to take off points and then justify why I took off those points. It's easier just to give points and move on. (And if anybody wants to criticize this, I'll be happy to launch into a diatribe on the psychometric dumpster fire that most assignments and their associated grading scales really are.)

Also prof: me too. I'm much more likely to provide comments on the first couple of exams I grade than on the later ones.

I've found that gradescope is helpful in this regard, because it at least forces every point assignment to be matched to a rubric item. I don't have data, but I believe it makes our grading a lot more uniform compared to the pre-gradescope days. (This might be easier in grading computer science exams than in more subjective areas, though.)

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#32

Post by dboreham » Sat, Apr 20, 2024, 2:38 PM UTC

Again, if this kind of thing surprises you, read this book: https://www.amazon.com/Knowledge-Illusion-Never-Think-Alone/... The human brain is just a fancy ChatGPT with an internal UI that fools itself into believing it is more logical/smart than it actually is.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#33

Post by llm_trw » Sat, Apr 20, 2024, 2:39 PM UTC

I'm willing to take bets that in 15 years there will be a scandal about faked data by at least one of the researches in this paper.

It smell just like every other interesting psychology result that at best is a fluke.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#34

Post by bigfudge » Sat, Apr 20, 2024, 2:39 PM UTC

bluGill wrote: post #10

buggy6257 wrote: post #4

Earlier quoted context omitted.

Enforced randomization isn't going to fix the problem, it just evenly distributes the problem. Based on these results, it would mean that the graders are just getting tired/lazy/inattentive the further they get in their stack of papers to grade. That's the problem the needs to be fixed, not the order they get graded in. Enforced randomization is simply a short term alleviation so no student(s) get disproportionately…

> it would mean that the graders are just getting tired/lazy/inattentive the further they get in their stack of papers Or maybe they are getting better / more picky. I know in code reviews I often pass a few and then notice something that I realize was also wrong in previous reviews I allowed, but later reviews that day (week?) will not allow that.

In my case, I have to make a conscious effort to remain consistently (in)tolerant of lazy writing. It’s hard to keep on reading between the lines and giving the benefit of the doubt.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#35

Post by rjzzleep » Sat, Apr 20, 2024, 2:39 PM UTC

bluGill wrote: post #10

buggy6257 wrote: post #4

Earlier quoted context omitted.

Enforced randomization isn't going to fix the problem, it just evenly distributes the problem. Based on these results, it would mean that the graders are just getting tired/lazy/inattentive the further they get in their stack of papers to grade. That's the problem the needs to be fixed, not the order they get graded in. Enforced randomization is simply a short term alleviation so no student(s) get disproportionately…

> it would mean that the graders are just getting tired/lazy/inattentive the further they get in their stack of papers Or maybe they are getting better / more picky. I know in code reviews I often pass a few and then notice something that I realize was also wrong in previous reviews I allowed, but later reviews that day (week?) will not allow that.

I had the same conclusion. You learn things as you go, including things you don't like.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#36

Post by zeroonetwothree » Sat, Apr 20, 2024, 2:40 PM UTC

BugsJustFindMe wrote: post #26

buggy6257 wrote: post #4

Earlier quoted context omitted.

Enforced randomization isn't going to fix the problem, it just evenly distributes the problem. Based on these results, it would mean that the graders are just getting tired/lazy/inattentive the further they get in their stack of papers to grade. That's the problem the needs to be fixed, not the order they get graded in. Enforced randomization is simply a short term alleviation so no student(s) get disproportionately…

> Enforced randomization isn't going to fix the problem, it just evenly distributes the problem. Evenly distributing the problem does fix the problem. Proportionality is what matters. Grading being arbitrary is fine if everyone is graded equally.

Random order would still mean a few students in the class get unlucky and near the end the majority of the time. Although over the course of all classes it would tend to even out somewhat.

It’s certainly better than fixed order.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#37

Post by 13of40 » Sat, Apr 20, 2024, 2:40 PM UTC

bluGill wrote: post #10

buggy6257 wrote: post #4

Earlier quoted context omitted.

Enforced randomization isn't going to fix the problem, it just evenly distributes the problem. Based on these results, it would mean that the graders are just getting tired/lazy/inattentive the further they get in their stack of papers to grade. That's the problem the needs to be fixed, not the order they get graded in. Enforced randomization is simply a short term alleviation so no student(s) get disproportionately…

> it would mean that the graders are just getting tired/lazy/inattentive the further they get in their stack of papers Or maybe they are getting better / more picky. I know in code reviews I often pass a few and then notice something that I realize was also wrong in previous reviews I allowed, but later reviews that day (week?) will not allow that.

I've participated in day-long and multi-day interview events for job candidates, and I see the same effect. At the beginning you don't have a frame of reference and you're more likely to question your own decision or give someone the benefit of the doubt, but by the end you're far more systematic, plus a little bit numb to the effect your decision is having.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#38

Post by exe34 » Sat, Apr 20, 2024, 2:40 PM UTC

tokai wrote: post #5

>One simple fix would be to make random order the default setting. Fixed in the sense that the bias will be random. Presumably students graded last will still receive lower grades.

It should average out over their career at the university - whereas if the alphabetical order is kept, then they would be systematically penalised.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#39

Post by [deleted] » Sat, Apr 20, 2024, 2:41 PM UTC

bluGill wrote: post #10

buggy6257 wrote: post #4

Earlier quoted context omitted.

Enforced randomization isn't going to fix the problem, it just evenly distributes the problem. Based on these results, it would mean that the graders are just getting tired/lazy/inattentive the further they get in their stack of papers to grade. That's the problem the needs to be fixed, not the order they get graded in. Enforced randomization is simply a short term alleviation so no student(s) get disproportionately…

> it would mean that the graders are just getting tired/lazy/inattentive the further they get in their stack of papers Or maybe they are getting better / more picky. I know in code reviews I often pass a few and then notice something that I realize was also wrong in previous reviews I allowed, but later reviews that day (week?) will not allow that.

[deleted]

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#40

Post by furyofantares » Sat, Apr 20, 2024, 2:41 PM UTC

buggy6257 wrote: post #4

hilux wrote: post #2

> Wang noted that for a small group of graders (about 5%) that grade from Z to A, the grade gap flips as expected This is critical. Otherwise we could not discount some group (e.g. some ethnicity) disproportionately occupying one end of the alphabet or another. Super interesting and important finding. I hope this gets wide visibility and universities take a break from politicking to fix the problem - presumably throu…

Enforced randomization isn't going to fix the problem, it just evenly distributes the problem. Based on these results, it would mean that the graders are just getting tired/lazy/inattentive the further they get in their stack of papers to grade. That's the problem the needs to be fixed, not the order they get graded in. Enforced randomization is simply a short term alleviation so no student(s) get disproportionately…

It's a 0.6 gap from top to bottom out of a score of 100. Plus or minus a third of a percent from average. Pretty small effect. But it would add up (or, well, persist - it wouldn't get bigger) if it happens to you for every assignment for every class and that sucks.

If there's more than one assignment you can basically erase it by randomizing each separately.

If you really care beyond that then randomize for one assignment, flip it for the next, then randomize again for the next etc.