Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Flawed Algorithms Are Grading Millions of Students’ Essays

vice.com

Post reply on HN
Print view View original

141–150 of 302 posts

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#141

Post by ssivark »

jkdufair wrote: post #122
mlthoughts2018 wrote: post #101

Earlier quoted context omitted.

In production problems that I work on, we don’t even really use things from within the past year. These problems are just incredibly well-solved with fairly vanilla LSTM networks from 2-3 years ago. Enough so that while it’s probably premature for fully automated essay grading, it’s not _crazy_ to make a product from models trained to solve this problem.

I have a grant where were are doing just that. Implementing more or less SOTA research using fairly vanilla LSTM networks from 2-3 years ago (primarily Taghipour & Ng) to provide low stakes feedback to students on their essays in one of our teaching tools at Purdue. It’s based on research using the Kaggle ASAP database and we have found it to be pretty accurate across a variety of domains in early testing. Though som…

Very interesting. Are there any meta-analyses / reviews that summarize progress in this area? Would it be possible to share your grant proposal -- I'd be curious to get an idea of what is being attempted.

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#142

Post by daveFNbuck »

dahart wrote: post #133

> Utah has been using AI as the primary scorer on its standardized tests for several years. “It was a major cost to our state to hand score, in addition to very time consuming,” said Cydnee Carter, the state’s assessment development coordinator. The automated process also allowed the state to give immediate feedback to students and teachers, she said. Yes, education takes time and costs money. Yes, not educating is b…

Do you just get to keep submitting the essay to see what score it will get before you turn it in? That sounds like a bigger problem than any of the particulars about how the grading is done.

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#143

Post by nyxtom »

dahart wrote: post #133

> Utah has been using AI as the primary scorer on its standardized tests for several years. “It was a major cost to our state to hand score, in addition to very time consuming,” said Cydnee Carter, the state’s assessment development coordinator. The automated process also allowed the state to give immediate feedback to students and teachers, she said. Yes, education takes time and costs money. Yes, not educating is b…

This needs to just be outright banned

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#144

Post by carapace »

danenania wrote: post #81
Spivak wrote: post #26

Earlier quoted context omitted.

In a forum of CS people I'm surprised this is one of the top opinions. Our field is full of super surprising results like this -- that you don't have to actually understand the text at beyond basic grammar structures to reasonably accurately predict the score a human would give it. Like this kind of thing should be cool , not insane. I mean wasn't it cool in your AI class when you learned that DFS could play Mario if…

"...that you don't have to actually understand the text at beyond basic grammar structures to reasonably accurately predict the score a human would give it" That only really shows that the humans they're training on are terrible at grading essays.

Thank you.

GIGO is our God now.

https://en.wikipedia.org/wiki/Garbage_in%2C_garbage_out

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#145

Post by killjoywashere »

VikingCoder wrote: post #15

My mother worked grading standardized tests. It was a hellish job for many reasons (limited breaks, etc.) One question she had to grade was essentially, "What's something you want your teacher to know about you?" It was an essay answer, and she was supposed to grade it on grammar, etc. Just the mechanical aspects of writing. (The real question explained the details more, but that was the core of the question.) She sa…

Tell you mom to take those numbers and the company's name to the police. They can walk back the identification problem.

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#146

Post by drdeca »

DATACOMMANDER wrote: post #124
bendbro wrote: post #110

Earlier quoted context omitted.

Do you suspect this is a one off mistake or do you think you make repeated mistakes of this kind? I'm always worried I'm doing some particular thing, interpersonally, repeatedly, but I fail to notice.

That’s what you’re doing wrong.

what?

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#147

Post by dahart »

daveFNbuck wrote: post #142
dahart wrote: post #133

> Utah has been using AI as the primary scorer on its standardized tests for several years. “It was a major cost to our state to hand score, in addition to very time consuming,” said Cydnee Carter, the state’s assessment development coordinator. The automated process also allowed the state to give immediate feedback to students and teachers, she said. Yes, education takes time and costs money. Yes, not educating is b…

Do you just get to keep submitting the essay to see what score it will get before you turn it in? That sounds like a bigger problem than any of the particulars about how the grading is done.

In this case, there was a limit to the number of times the essay could be submitted, and there was a required score that needed to be obtained within that limit, otherwise the grade would go down. The limit was something like 20 tries, and when I got there they’d already used maybe 14 of them.

I could perhaps see value in having unlimited tries, as a teaching aid, if the result wasn’t being used for grading. That would at least leave room for curiosity and exploration. And, more importantly, I could see value if the software wasn’t essentially a scam that fundamentally is not able do what is advertised. If the software really could grade essays reliably, and provide meaningful suggestions for improvement, then maybe it could be used to help educate students, in conjunction with the teacher’s guidance. But the software does not grade reliably, and it absolutely does not offer meaningful constructive feedback, and the teachers were using it to avoid reading essays, not to supplement their own expertise.

One of the several amusing ironies here is how the software company has convinced the state and teachers to willingly replace themselves with bots, despite obvious evidence that the humans can do the job better.

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#148

Post by aasasd »

bendbro wrote: post #110
shaggyfrog wrote: post #48

Earlier quoted context omitted.

Punch up, not down.

Do you suspect this is a one off mistake or do you think you make repeated mistakes of this kind? I'm always worried I'm doing some particular thing, interpersonally, repeatedly, but I fail to notice.

I embraced the awkwardness, and now I know for sure that I'm doing ‘a thing,’ instead of worrying about whether I do or not.

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#149

Post by c3534l »

dahart wrote: post #133

> Utah has been using AI as the primary scorer on its standardized tests for several years. “It was a major cost to our state to hand score, in addition to very time consuming,” said Cydnee Carter, the state’s assessment development coordinator. The automated process also allowed the state to give immediate feedback to students and teachers, she said. Yes, education takes time and costs money. Yes, not educating is b…

It's be easier and equally fair to just grade student's essays by rolling a pair of dice.
Post reply on HN