Most exam grading is not viewing the writing as a whole but rather looking for incidences of specific points to assign credit for. One could imagine an LLM be quite effective at labeling sentences as pertaining to a predefined idea at scale.
U-M finds students with alphabetically lower-ranked names receive lower grades
21–30 of 284 posts
Re: U-M finds students with alphabetically lower-ranked names receive lower grades
#22> Wang noted that for a small group of graders (about 5%) that grade from Z to A, the grade gap flips as expected This is critical. Otherwise we could not discount some group (e.g. some ethnicity) disproportionately occupying one end of the alphabet or another. Super interesting and important finding. I hope this gets wide visibility and universities take a break from politicking to fix the problem - presumably throu…
Enforced randomization isn't going to fix the problem, it just evenly distributes the problem. Based on these results, it would mean that the graders are just getting tired/lazy/inattentive the further they get in their stack of papers to grade. That's the problem the needs to be fixed, not the order they get graded in. Enforced randomization is simply a short term alleviation so no student(s) get disproportionately…
Even distribution would fix the problem. If grading has a subjective component, there will always be deviations from the "correct" grade. If those patterns are randomly distributed over all students, their grade averages will be comparable again.
Re: U-M finds students with alphabetically lower-ranked names receive lower grades
#23>“We kind of suspect that fatigue is one of the major factors that is driving this effect, because when you’re working on something for a long period of time, you get tired and then you start to lose your attention and your cognitive abilities are dropping,” Pei said. there is a similar effect found here https://en.wikipedia.org/wiki/Hungry_judge_effect
I believe the hungry judge effect has generally been accepted as false.
Re: U-M finds students with alphabetically lower-ranked names receive lower grades
#24Job interviews have similar effects
Re: U-M finds students with alphabetically lower-ranked names receive lower grades
#25As someone whose initials are Z and W, I tend to notice alpha sort a lot. Asking a friend whose initials are A and B about this, it's not something they ever noticed. I haven't noticed a grading/ranking difference, but far more frequently I'll hear that "oh, we ran out of item/time/etc. before we got to you", which has made me much more sensitive to issues of planning/organization.
Like most inequities, those who are in the benefiting group frequently don't realize that privilege.
Re: U-M finds students with alphabetically lower-ranked names receive lower grades
#26> Wang noted that for a small group of graders (about 5%) that grade from Z to A, the grade gap flips as expected This is critical. Otherwise we could not discount some group (e.g. some ethnicity) disproportionately occupying one end of the alphabet or another. Super interesting and important finding. I hope this gets wide visibility and universities take a break from politicking to fix the problem - presumably throu…
Enforced randomization isn't going to fix the problem, it just evenly distributes the problem. Based on these results, it would mean that the graders are just getting tired/lazy/inattentive the further they get in their stack of papers to grade. That's the problem the needs to be fixed, not the order they get graded in. Enforced randomization is simply a short term alleviation so no student(s) get disproportionately…
> Enforced randomization isn't going to fix the problem, it just evenly distributes the problem.
Evenly distributing the problem does fix the problem. Proportionality is what matters. Grading being arbitrary is fine if everyone is graded equally.
Re: U-M finds students with alphabetically lower-ranked names receive lower grades
#27> Wang noted that for a small group of graders (about 5%) that grade from Z to A, the grade gap flips as expected This is critical. Otherwise we could not discount some group (e.g. some ethnicity) disproportionately occupying one end of the alphabet or another. Super interesting and important finding. I hope this gets wide visibility and universities take a break from politicking to fix the problem - presumably throu…
Enforced randomization isn't going to fix the problem, it just evenly distributes the problem. Based on these results, it would mean that the graders are just getting tired/lazy/inattentive the further they get in their stack of papers to grade. That's the problem the needs to be fixed, not the order they get graded in. Enforced randomization is simply a short term alleviation so no student(s) get disproportionately…
I was a grader once. I guarantee if someone gives a good answer they'll get full marks even near the bottom of the stack. For BS answers I'll admit I got less generous as the hours went on.
No one's getting hurt by this system if it's randomized. It's a matter of graders giving out partial credit for wrong answers which is discretionary. Rarely students are granted a small mercy. Seems OK.
Re: U-M finds students with alphabetically lower-ranked names receive lower grades
#28> Wang noted that for a small group of graders (about 5%) that grade from Z to A, the grade gap flips as expected This is critical. Otherwise we could not discount some group (e.g. some ethnicity) disproportionately occupying one end of the alphabet or another. Super interesting and important finding. I hope this gets wide visibility and universities take a break from politicking to fix the problem - presumably throu…
Enforced randomization isn't going to fix the problem, it just evenly distributes the problem. Based on these results, it would mean that the graders are just getting tired/lazy/inattentive the further they get in their stack of papers to grade. That's the problem the needs to be fixed, not the order they get graded in. Enforced randomization is simply a short term alleviation so no student(s) get disproportionately…
As the number of assignments grows, wouldn’t randomization help converge on the more accurate grades (in aggregate)?
Re: U-M finds students with alphabetically lower-ranked names receive lower grades
#29> Wang noted that for a small group of graders (about 5%) that grade from Z to A, the grade gap flips as expected This is critical. Otherwise we could not discount some group (e.g. some ethnicity) disproportionately occupying one end of the alphabet or another. Super interesting and important finding. I hope this gets wide visibility and universities take a break from politicking to fix the problem - presumably throu…
Enforced randomization isn't going to fix the problem, it just evenly distributes the problem. Based on these results, it would mean that the graders are just getting tired/lazy/inattentive the further they get in their stack of papers to grade. That's the problem the needs to be fixed, not the order they get graded in. Enforced randomization is simply a short term alleviation so no student(s) get disproportionately…
For me I grade tests as follows. The stack is created as students turn in the test. I grade the first page in that order. The stack reverses for the second page. So on and so forth. I teach college math. I just cant imagine a system of grading done in alphabetical order.
Re: U-M finds students with alphabetically lower-ranked names receive lower grades
#30Earlier quoted context omitted.
Enforced randomization isn't going to fix the problem, it just evenly distributes the problem. Based on these results, it would mean that the graders are just getting tired/lazy/inattentive the further they get in their stack of papers to grade. That's the problem the needs to be fixed, not the order they get graded in. Enforced randomization is simply a short term alleviation so no student(s) get disproportionately…
I was a grader once. I guarantee if someone gives a good answer they'll get full marks even near the bottom of the stack. For BS answers I'll admit I got less generous as the hours went on. No one's getting hurt by this system if it's randomized. It's a matter of graders giving out partial credit for wrong answers which is discretionary. Rarely students are granted a small mercy. Seems OK.
>For BS answers I'll admit I got less generous as the hours went on.
What do you think is the cause of this? Do you become more cynical (and less generous) because you’ve seen so many BS answers previously? Is it just that getting fatigued makes you less generous?