Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

U-M finds students with alphabetically lower-ranked names receive lower grades

record.umich.edu

Post reply on HN
Print view View original

51–60 of 284 posts

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#51

Post by zeroonetwothree »

princeb wrote: post #16

>“We kind of suspect that fatigue is one of the major factors that is driving this effect, because when you’re working on something for a long period of time, you get tired and then you start to lose your attention and your cognitive abilities are dropping,” Pei said. there is a similar effect found here https://en.wikipedia.org/wiki/Hungry_judge_effect

The thing is, it’s unclear why that effect would make you give people lower grades. surely an equally reasonable guess is that less cognitive abilities could make you give higher grades because you don’t notice errors?

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#53

Post by zeroonetwothree »

boesboes wrote: post #3

This seems related: https://news.ycombinator.com/item?id=39672111 As in, order matters

This looks like one of the classic studies that won’t reproduce. For one thing, the effect size is unreasonably large. 50% more positive words just because of sequence order would be so huge we should be able to notice it anecdotally.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#54

Post by falseprofit »

bumby wrote: post #28
buggy6257 wrote: post #4

Earlier quoted context omitted.

Enforced randomization isn't going to fix the problem, it just evenly distributes the problem. Based on these results, it would mean that the graders are just getting tired/lazy/inattentive the further they get in their stack of papers to grade. That's the problem the needs to be fixed, not the order they get graded in. Enforced randomization is simply a short term alleviation so no student(s) get disproportionately…

As the number of assignments grows, wouldn’t randomization help converge on the more accurate grades (in aggregate)?

It would help, but with only a couple dozen courses and most determined by a couple exams it’s not quite a large number.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#56

Post by BugsJustFindMe »

zeroonetwothree wrote: post #36
BugsJustFindMe wrote: post #26

Earlier quoted context omitted.

> Enforced randomization isn't going to fix the problem, it just evenly distributes the problem. Evenly distributing the problem does fix the problem. Proportionality is what matters. Grading being arbitrary is fine if everyone is graded equally.

Random order would still mean a few students in the class get unlucky and near the end the majority of the time. Although over the course of all classes it would tend to even out somewhat. It’s certainly better than fixed order.

"randomization" is not the important part here. "evenly distributing" is. It is absolutely possible to reorder the sequence fairly such that your scenario doesn't occur. It could even to a human observer look randomized if you want. In a trivial example case where the effect were linear you could just switch the order back and forth, and on average every student would receive the same middle-of-group impact.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#57

Post by zeroonetwothree »

dboreham wrote: post #32

Again, if this kind of thing surprises you, read this book: https://www.amazon.com/Knowledge-Illusion-Never-Think-Alone/... The human brain is just a fancy ChatGPT with an internal UI that fools itself into believing it is more logical/smart than it actually is.

If anything the difference only being 0.6% seems pretty impressive for the brain.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#58

Post by furyofantares »

zeroonetwothree wrote: post #45
exe34 wrote: post #38

Earlier quoted context omitted.

It should average out over their career at the university - whereas if the alphabetical order is kept, then they would be systematically penalised.

It won’t average out perfectly. There will still be lucky and unlucky students. Of course it’s better than a fixed order, and if it’s easy to switch then might as well. But we should keep thinking about how we can make it even better.

Since the effect looks very small, it looks to me like it's only a problem because it adds up if it happens for every assignment for every course. I don't think it needs to average out perfectly; it looks to me like you'd have to be astronomically lucky/unlucky for it to matter if each assignment is in random order.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#60

Post by zeroonetwothree »

jimmar wrote: post #13

Order effects are real. I'm a prof. I notice that the longer I grade, the less motivated I am to take off points and then justify why I took off those points. It's easier just to give points and move on. (And if anybody wants to criticize this, I'll be happy to launch into a diatribe on the psychometric dumpster fire that most assignments and their associated grading scales really are.)

This is the opppsite of the effect they found. I do wonder if there is a big difference depending on grader and the study found some kind of average.
Post reply on HN