Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Flawed Algorithms Are Grading Millions of Students’ Essays

vice.com

Post reply on HN
Print view View original

61–70 of 302 posts

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#61

Post by jedberg »

Any state that relies on the AI as the primary grader does not understand the current state of AI.

It would make sense to use the AI as a first pass, and then not randomly grade the essays with a human, but specifically choose all the essays that are on the cusp of the pass fail line. Then use all those human generated scores to update the model, especially if someone moves from pass to fail or fail to pass. Then maybe throw in a few of the really high and really low outliers to make sure those are right, and throw away your entire model if the human scores are drastically different (and obviously don't tell the humans what the computer score was so they have no idea if they're reading a "cusp" essay or an outlier essay).

But putting the educational fate (and therefore future earnings) in the hands of an AI is unconscionable.

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#62

Post by snazz »

Meekro wrote: post #40

From the article: " All essays scored by E-rater are also graded by a human and discrepancies are sent to a second human for a final grade. Because of that system, ETS does not believe any students have been adversely affected by the bias detected in E-rater."

That particular company seems to do a not-horrible job. But they’re not the only game in town, so presumably most or many essays are graded by another company’s system.

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#63

Post by cmroanirgo »

Spivak wrote: post #26
drngdds wrote: post #9

This is my first time learning that AI-graded essays are a thing. Am I the only one who thinks that's insane? I feel like you'd probably have to have an AGI to meaningfully evaluate an essay.

In a forum of CS people I'm surprised this is one of the top opinions. Our field is full of super surprising results like this -- that you don't have to actually understand the text at beyond basic grammar structures to reasonably accurately predict the score a human would give it. Like this kind of thing should be cool , not insane. I mean wasn't it cool in your AI class when you learned that DFS could play Mario if…

I came first in English for my school, many moons ago. Leading up to the finals, I regularly finished ahead of the hard core the English essay people, generally to my amusement. My exam essay responses were generally half the length (sometimes even shorter) than the prodigious writers. Although I've an ok vocabulary, I always made sure I made the right choice of word to hit a specific meaning, rather than choosing words with a high syllable count.

I'd find it highly interesting to see what kind of result I'd get using an automated system.

Why?

Because, I once asked a teacher (also an examiner) why I got good grades above the others, and the answer surprised me: my answers were generally unique /refreshingly different, to the point/ not too long and easy to read.

I suspect with this new system, I'd be an average student. It'd also be interesting to find out, several years down the road, if the automated system could be gamed at all -- I suspect it could, and teachers would help students 'maximise' their scores as a result of that.

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#64

Post by shkkmo »

topologistics wrote: post #47
Hitton wrote: post #27

Earlier quoted context omitted.

>So the takeaway is that if you’re one of the 80-95% of (typically black or female) people who the machine scored dramatically lower It's quite funny how are some people manipulated to think that society and especially education is somehow biased against minorities or women when opposite is true[1][2]. [1] https://www.studyinternational.com/news/record-high-numbers-... [2] https://www.independent.co.uk/news/world/ame…

The idea that black folks and females score dramatically lower on anything is a commonly held racist opinion that is foundational for the continuation of institutional oppression. When you assume the worst about people it ends up hindering their personal progress.

They do when graded by a computer program as described in the article. Are you disputing the article's veracity? What is your point?

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#65

Post by mrep »

VikingCoder wrote: post #39
rhexs wrote: post #20

Earlier quoted context omitted.

Or...report it to the police? I’d gladly risk my job to do the right thing in that instance.

What my mom saw had an ID number on it. No other demographics. And she was grading from multiple states. So do what? Contact her local police? With a written accusation from a child? Is that enough to get a warrant to force the company to release the demographic information? And people don't work at a job like that because they want to. They work there because they need the money. Everything she took in and out of th…

> So do what? Contact her local police?

> With a written accusation from a child? Is that enough to get a warrant to force the company to release the demographic information?

Absolutely! My girlfriend works as a counselor at a school and she is required by law to report all serious abuses by parents.

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#66

Post by mxcrossb »

harry8 wrote: post #34
VikingCoder wrote: post #15

My mother worked grading standardized tests. It was a hellish job for many reasons (limited breaks, etc.) One question she had to grade was essentially, "What's something you want your teacher to know about you?" It was an essay answer, and she was supposed to grade it on grammar, etc. Just the mechanical aspects of writing. (The real question explained the details more, but that was the core of the question.) She sa…

Some of these will be 100% true as well. But don't make the mistake that there are no kids who go for shock value or are wantonly manipulative when they know it can't come back to them. So how many are true and how many false? I have no clue. Literally none. And no it doesn't make me feel any better about the screams of existential agony even if that were a low percentage. Could be high too.

I don’t know about calling it manipulative. I remember taking the ACT, and struggling to plan out one of my essays. It was something like “tell us about a book that inspired you”. So I changed details about the plot so it all fit nicely and was easy to write. I can see something similar here, where someone takes on a persona when writing in order to effectively communicate.

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#67

Post by shkkmo »

Meekro wrote: post #40

From the article: " All essays scored by E-rater are also graded by a human and discrepancies are sent to a second human for a final grade. Because of that system, ETS does not believe any students have been adversely affected by the bias detected in E-rater."

Also from the article:

> Of those 21 states, three said every essay is also graded by a human. But in the remaining 18 states, only a small percentage of students’ essays—it varies between 5 to 20 percent—will be randomly selected for a human grader to double check the machine’s work.

So that applies only in a minority of cases.

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#68

Post by wesammikhail »

shaggyfrog wrote: post #48
empath75 wrote: post #28

Earlier quoted context omitted.

‘Poor sentence structure and grammar, 1 point out of five. Sorry your daddy touches you.’

Punch up, not down.

boy do I hate that saying. It literally adds nothing to the conversation!

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#69

Post by danharaj »

mynameishere wrote: post #60
VikingCoder wrote: post #15

My mother worked grading standardized tests. It was a hellish job for many reasons (limited breaks, etc.) One question she had to grade was essentially, "What's something you want your teacher to know about you?" It was an essay answer, and she was supposed to grade it on grammar, etc. Just the mechanical aspects of writing. (The real question explained the details more, but that was the core of the question.) She sa…

I wonder if your mother scored those higher than she might have. And I wonder if that's why such essays were written. Hmm, wonder...

Why did you post this?
Post reply on HN