81–90 of 284 posts
Let’s just hope parents don’t try to game the system by starting to name their kids AAAi Aung.
>“We kind of suspect that fatigue is one of the major factors that is driving this effect, because when you’re working on something for a long period of time, you get tired and then you start to lose your attention and your cognitive abilities are dropping,” Pei said. there is a similar effect found here https://en.wikipedia.org/wiki/Hungry_judge_effect
The thing is, it’s unclear why that effect would make you give people lower grades. surely an equally reasonable guess is that less cognitive abilities could make you give higher grades because you don’t notice errors?
Earlier quoted context omitted.
Enforced randomization isn't going to fix the problem, it just evenly distributes the problem. Based on these results, it would mean that the graders are just getting tired/lazy/inattentive the further they get in their stack of papers to grade. That's the problem the needs to be fixed, not the order they get graded in. Enforced randomization is simply a short term alleviation so no student(s) get disproportionately…
> it would mean that the graders are just getting tired/lazy/inattentive the further they get in their stack of papers Or maybe they are getting better / more picky. I know in code reviews I often pass a few and then notice something that I realize was also wrong in previous reviews I allowed, but later reviews that day (week?) will not allow that.
Additionally, universities (and, by extension departments) want grades to approximately follow a normal distribution (and yes, you in the back, their actions show they do actually want that, even if they say otherwise).
When you start grading a problem you have some idea what a "good" solution looks like, what an "ok" solution looks like, and same for "bad" solutions... If you award points based on that, the result will be a normal-ish distribution. But your idea of a good/ok/bad solution evolves as you see more papers.
There's two reasons for that:
First, you can't (ahead of time) imagine all the ways that students will invent to fuck up a problem set, and find edge cases in your grading rubric that result in unfairly-high or -low scores. As you gain experience teaching, you will anticipate more of the ways, but you will never anticipate every way.
Second, the TA/grader wants to be able to stack-rank the papers and have the scores be monotonic. The grader wants this because non-monotonic scoring triggers far more complaining than harsh scoring or picky scoring. When you come across papers that are worse than ones you've already recently graded, you assign even lower scores.
This results in a ratcheting effect with more extreme scores as you get closer to the bottom of the pile. But, since the mean score is usually a B/B-/C+ (~75-85), and since scores are usually limited to the range 0-100, this means that papers closer to the bottom will receive statistically lower scores.
Now, you could go back a re-grade ones you've already done, but:
1. The university is officially only paying you for 20hrs/week (and requires a signed end-of-semester statement attesting to the same).
2. The assigned workload of teaching and grading doesn't permit a two-pass grading scheme while keeping within 20 hours.
3. If you complain to the graduate ombudsman about the workload needing more than 20 hours, you won't have funding next semester (so you have a prisoner's dilemma among TAs who might want to grade more fairly).
4. If you're grading (say) a final exam for a frosh/soph class, you're probably in a room with 4-8 other graders late into the night. One effective way to make your coworkers hate you is to be that guy who always finishes grading his stack last, when everyone is worried about catching the last train/bus.
Basically, all the incentives are aligned to make this happen.
> Wang noted that for a small group of graders (about 5%) that grade from Z to A, the grade gap flips as expected This is critical. Otherwise we could not discount some group (e.g. some ethnicity) disproportionately occupying one end of the alphabet or another. Super interesting and important finding. I hope this gets wide visibility and universities take a break from politicking to fix the problem - presumably throu…
Order effects are real. I'm a prof. I notice that the longer I grade, the less motivated I am to take off points and then justify why I took off those points. It's easier just to give points and move on. (And if anybody wants to criticize this, I'll be happy to launch into a diatribe on the psychometric dumpster fire that most assignments and their associated grading scales really are.)
This is the opppsite of the effect they found. I do wonder if there is a big difference depending on grader and the study found some kind of average.
> The researchers collected available historical data of all programs, students and assignments on Canvas from the fall 2014 semester to the summer 2022 semester.
Thousands of students X 8 years X lots of assignments per year and you get a sample size so big that it would be hard not to find statistically significant effects.
So I guess "alphabetically lower ranked" means the last letters of the alphabet, not first? Confused.
I work in academia. When we grade exams, the order of the exams on the stack is the order in which they were collected in the room (people can sit wherever they like). For grading, we are usually 5 people in a single room, and everyone grades a specific exercise for consistency. The exams are getting shuffled heavily , with everyone just grabbing stacks, looking for exams where "their" exercise was not yet graded, an…
I searched the authors in google scholar but I couldn't find it.
As someone whose initials are Z and W, I tend to notice alpha sort a lot. Asking a friend whose initials are A and B about this, it's not something they ever noticed. I haven't noticed a grading/ranking difference, but far more frequently I'll hear that "oh, we ran out of item/time/etc. before we got to you", which has made me much more sensitive to issues of planning/organization.
Like most inequities, those who are in the benefiting group frequently don't realize that privilege.