Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

U-M finds students with alphabetically lower-ranked names receive lower grades

record.umich.edu

Post reply on HN
Print view View original

71–80 of 284 posts

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#71

Post by cyanydeez »

13of40 wrote: post #37
bluGill wrote: post #10

Earlier quoted context omitted.

> it would mean that the graders are just getting tired/lazy/inattentive the further they get in their stack of papers Or maybe they are getting better / more picky. I know in code reviews I often pass a few and then notice something that I realize was also wrong in previous reviews I allowed, but later reviews that day (week?) will not allow that.

I've participated in day-long and multi-day interview events for job candidates, and I see the same effect. At the beginning you don't have a frame of reference and you're more likely to question your own decision or give someone the benefit of the doubt, but by the end you're far more systematic, plus a little bit numb to the effect your decision is having.

For grading, you could probably just add a mediating factor and throw in test cases that calibrate the factor and then you curve everyone on that factor.

It'd seemingly be more work but would result in averages that are more reasonable to the changes in stress.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#72

Post by [deleted] »

hilux wrote: post #2

> Wang noted that for a small group of graders (about 5%) that grade from Z to A, the grade gap flips as expected This is critical. Otherwise we could not discount some group (e.g. some ethnicity) disproportionately occupying one end of the alphabet or another. Super interesting and important finding. I hope this gets wide visibility and universities take a break from politicking to fix the problem - presumably throu…

[deleted]

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#73

Post by bee_rider »

dotnet00 wrote: post #9

Yep, I noticed this with myself too when I first did some grading a few months ago. There was also the factor that the ones I graded initially did not make certain mistakes or answered in expected ways, such that when I did encounter unexpected answers/mistakes, I had to go back and rethink the grading on the papers I had graded previously. Eg if someone answered in a way that made me think an answer I considered inc…

I especially noticed this when grading programming projects, because it is slightly complicated.

I’d either find that:

A bug was really common, got to re-evaluate after the first couple times I see it, apparently it is an easy mistake to make.

Or, I’d find a new bug that was pretty common, but which I didn’t know about at first. Got to update my tests and re-run everybody.

I tended to be really thorough and re-do the whole stack eventually, but it was a real pain. Could have half-assed it of course, but they spend weeks on these things, feel like I owe them honest feedback.

It would tend to lead me to “softer” grading as well, if you are lazy and only check for a couple bugs, you might take a large number of points off for each problem. Finding some problems and punishing them harshly is not very fair for those students that randomly hit the bugs you expect. If you find every bug, you can only take a couple points off per bug without tanking everybody’s score.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#74

Post by analog31 »

I propose one of the following:

1. Keep the present system of grading by alphabetical order

2. Record the order in which the papers are actually graded

When the grading is done, the teacher assigns a point scale (A = 90, B = 80 or whatever) but the computer does a regression fit and removes the bias.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#76

Post by zeroonetwothree »

xyst wrote: post #62

That 0.6 pt gap over multiple semesters is the difference between graduating with “summa cum laude” or “magma cum laude”

It’s 0.6% so it would only be if you happened to drop a letter grade as a result. Like 90.5 -> 89.9. And that would have to happen multiple times to significantly affect your GPA.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#77

Post by hilux »

llm_trw wrote: post #33

I'm willing to take bets that in 15 years there will be a scandal about faked data by at least one of the researches in this paper. It smell just like every other interesting psychology result that at best is a fluke.

The result seems pretty intuitive to me. The test is easy to re-run, unless the data have been "lost," which is not mentioned.

Most importantly, none of the researchers is a psychologist or behavioral economist or any kind of "social scientist."

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#78

Post by 1shooner »

This reminds me of an experience I had of just the opposite: tightly-controlled consistency in writing assessments:

Almost 20 years ago I worked for a standardized test essay grading service. We graded against all sorts of secondary-level rubrics (not AP, who do their own). These would usually be from 9 - 12 grade, from every US state, and evaluating everything from reading comprehension to subject matter-specific assessment. We'd do weeks long jobs of a single test (e.g. Alabama 9th grade reading proficiency). These usually had at least 3 dimensions, and at least 4 points per dimension. We would go through a week or more of training on a rubric, then another week of 'leveling', where a manager would occasionally bring you aside and talk through why that '3' you gave on a dimension should have been a '2'.

By the end of the training, we usually had had enough discussions and encountered enough edge cases to understand the weaknesses/inconsistencies in the rubric (which we had to abide by anyway). Once we were running at full-speed, everything was still double-graded and inconsistent scores were reviewed. Sometimes graders were pulled if they still didn't get the rubric.

It was a simultaneously stimulating and very boring job, and most readers were educators themselves. I wonder how long before it disappears completely.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#79

Post by ryandrake »

Maybe related, or maybe not, but I remember when I was in K-12 school back in the 80s and early 90s, they would always seat us physically in the class front-to-back by last name. So the kids with last names starting with A-D or so would always be in front, and the kids with last names starting with U-Z would be in the back. For every class. I remember this because many of my friends had last names "near" my last name since we were always in close proximity to each other. I vaguely remember, by the time we were in high school, there were definitely more high-achieving kids with A-D last names and definitely more of the troublemakers were U-Z. Was it caused by sitting in closer proximity to the teacher and getting more teacher attention? We'll never know because this wasn't an experiment and there wasn't a control group.

Re: U-M finds students with alphabetically lower-ranked names receive lower grades

#80

Post by ghaff »

noodlesUK wrote: post #6

At my university, almost all of our marking was pseudonymised. We were assigned a random candidate number at the beginning of each year, and that is what went on our important papers/exams. The less important coursework often didn’t bother with this, and used our student numbers instead, but the general idea was the same. We didn’t put our names on any of our work other than our dissertation (and a few trivial assign…

University exams, this probably makes a lot of sense. After all, the exam is the exam and whether a student is well-spoken and actively participates in class shouldn't matter for an exam grade. I'm less convinced that blinded conference proposals are a good idea--an argument I've had with various people. If you know based on past experience that someone will almost certainly hit a home run, I'm less inclined to pick a random person without obvious qualifications for the same topic--although just picking friends of the committee can obviously go too far.
Post reply on HN