Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Flawed Algorithms Are Grading Millions of Students’ Essays

vice.com

Post reply on HN
Print view View original

231–240 of 302 posts

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#231

Post by bryanrasmussen »

dagw wrote: post #227
raxxorrax wrote: post #211

Earlier quoted context omitted.

You have automated systems that rate essays without any human actually reading them? Kids, forget everything you know because crime does indeed pay off. Best grades will be reserved for those that try to cheat this system however it is implemented. Botting your essays is the way to go in the 21st century.

Given that the stack ranking at your future job will also be done by an "AI" (probably developed by the same company that graded you tests) this is a very useful skill to have.

Good point, I better teach my kid this skill so she can go on to a job programming one of these "AIs"

idiocracy will happen not because people get any stupider, but because the bots reward the stupid ones first.

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#232

Post by rwbcxrz »

ALittleLight wrote: post #157
dahart wrote: post #133

> Utah has been using AI as the primary scorer on its standardized tests for several years. “It was a major cost to our state to hand score, in addition to very time consuming,” said Cydnee Carter, the state’s assessment development coordinator. The automated process also allowed the state to give immediate feedback to students and teachers, she said. Yes, education takes time and costs money. Yes, not educating is b…

When I was a child I was obsessed with the "grade level" function in Microsoft Word. It was a preference you could enable on spell check to tell you the "grade level" of your writing. Every essay I wrote, I'd always force myself to reach the max "12.0" grade level. While writing I'd struggle over word choice, sentence structure, rearranging paragraphs, working on my tone etc, all in pursuit of the 12th grade way to p…

> Personally, I feel the solution to insufficient teacher time is to use peer grading much more, and spot checks. Get kids to read and revise each other's works frequently, and teachers should aim to grade at least N papers per student where N is much less than the number of papers a student writes.

That's how it's done in creative writing courses. I've always found it infinitely more helpful than only having feedback from the instructor, even if the instructor's feedback was generally more helpful/useful than peer feedback.

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#233

Post by mannykannot »

dagw wrote: post #183
piokoch wrote: post #181

Earlier quoted context omitted.

Frankly, I can't believe what I am reading. The idea that some "AI" grades essays automatically is idiotic and has nothing to do with education. Where is the place for discussions? Where is the place for ideas confrontation? Where is the place for writing style development? How this AI is supposed to grade things like repetitions (that can be either good rhetorical tool or a mistake, depending on context), etc? Who t…

My guess is that soon or later people will learn how to game that AI. Already been done. http://lesperelman.com/writing-assessment-robo-grading/babel... Here's a sample essay that is complete nonsense and got a perfect score on the GRE. http://lesperelman.com/wp-content/uploads/2015/12/6-6_ScoreI...

Professor Perelman had also previously demonstrated that this sort of scoring was going on when essays were scored by humans [1].

I suspect that, in addition to the scoring rules being written for speed and frugality, they were shaped by a poorly thought out attempt to make the scoring 'objective', and independent of the scorer's beliefs, attitudes and unconscious biases.

In one sense, this software (I will not call it 'AI') is an extension of all those bad ideas, only greatly amplified in a way that only software can.

In evolutionary biology, there is the concept of 'honest signalling'[2], a true and unfakable indicator, to a potential mate or predator, of an animal's fitness. That is what we are missing here.

[1] https://www.bostonglobe.com/opinion/2014/03/13/the-man-who-k...

[2] https://en.wikipedia.org/wiki/Signalling_theory

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#234

Post by Aromasin »

clay_the_ripper wrote: post #152
dahart wrote: post #133

> Utah has been using AI as the primary scorer on its standardized tests for several years. “It was a major cost to our state to hand score, in addition to very time consuming,” said Cydnee Carter, the state’s assessment development coordinator. The automated process also allowed the state to give immediate feedback to students and teachers, she said. Yes, education takes time and costs money. Yes, not educating is b…

Wow this is pretty shocking. I can understand using automated systems for something like math problems, it makes sense. There’s (usually) one right answer. But essays? This should be banned.

Having been forced to use an online math software for all my homework while at school, I vehemently disagree. It was so poor that it became a meme within my year group.

It would mark you as incorrect for using too many decimal places, even though it wouldn't tell you how many significant figures was required. I often remember it marking my answer as incorrect, even though it was identical to the answer they gave. Sometimes you'd have to show your working, but it couldn't handle brackets. Once I put the answer as "1+x=y" but they wanted the answer "y-1=x", and they marked it as incorrect.

I'm sure academic software design is leaps and bounds above what it was in the early 2000's, but to have a pupils futures hinge on what generally seems to be poorly tested code is dangerous.

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#235

Post by [deleted] »

jkdufair wrote: post #122
mlthoughts2018 wrote: post #101

Earlier quoted context omitted.

In production problems that I work on, we don’t even really use things from within the past year. These problems are just incredibly well-solved with fairly vanilla LSTM networks from 2-3 years ago. Enough so that while it’s probably premature for fully automated essay grading, it’s not _crazy_ to make a product from models trained to solve this problem.

I have a grant where were are doing just that. Implementing more or less SOTA research using fairly vanilla LSTM networks from 2-3 years ago (primarily Taghipour & Ng) to provide low stakes feedback to students on their essays in one of our teaching tools at Purdue. It’s based on research using the Kaggle ASAP database and we have found it to be pretty accurate across a variety of domains in early testing. Though som…

[deleted]

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#237

Post by bigred100 »

dahart wrote: post #133

> Utah has been using AI as the primary scorer on its standardized tests for several years. “It was a major cost to our state to hand score, in addition to very time consuming,” said Cydnee Carter, the state’s assessment development coordinator. The automated process also allowed the state to give immediate feedback to students and teachers, she said. Yes, education takes time and costs money. Yes, not educating is b…

This is a problem I have with a lot of human behavior. Instead of admitting you don’t have the resources to do something or aren’t willing to prioritize it, people come up with a bad version that’s not worth doing. Lots of things are worth doing poorly, but many of them I believe you just need to admit are not worth it unless a certain level of performance is met.

What’s even cheaper than AI? Tell the students to write some pages, have the teacher glance at the number of pages written, give full credit if the mark was met, and throw the papers out without reading them. It sounds like it would be similarly effective and less aggravating. Unfortunately, this would require humility on the part of the educators.

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#238

Post by mannykannot »

dcolkitt wrote: post #132
drngdds wrote: post #9

This is my first time learning that AI-graded essays are a thing. Am I the only one who thinks that's insane? I feel like you'd probably have to have an AGI to meaningfully evaluate an essay.

> I feel like you'd probably have to have an AGI to meaningfully evaluate an essay. So the reason this isn't the case, is because there are very simple metrics that tend to highly correlate with essay quality. It doesn't mean the grading-bot is actually evaluating essay quality. It's just looking for properties that are statistically associated with good essays. Remember, at the end of the day as long as the bot's ra…

Your last paragraph, and particularly the last sentence, epitomizes what is wrong with your whole thesis: the ultimate goal of the testing (and education itself, for that matter) is not to find people who can "do well on essays"; it is to develop analytical thinking.

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#239

Post by ironSkillet »

The underlying problem is that reading essays with a careful critical eye is not scalable. But another issue this highlights is the complete misalignment of incentives of the people who greenlit the adoption of this technology. Because educational outcomes are much harder to evaluate over the course of a bureaucrat's tenure than budget sizes (longer time horizon and many exogenous variables), there is a natural inclination to make decisions that reduce costs as long as they don't have any obvious (to them or their superiors) adverse outcome for students. This is a pretty low bar, especially so given that most bureaucrats do not have the background necessary to evaluate technical solutions.

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#240

Post by JoeAltmaier »

mannykannot wrote: post #238
dcolkitt wrote: post #132

Earlier quoted context omitted.

> I feel like you'd probably have to have an AGI to meaningfully evaluate an essay. So the reason this isn't the case, is because there are very simple metrics that tend to highly correlate with essay quality. It doesn't mean the grading-bot is actually evaluating essay quality. It's just looking for properties that are statistically associated with good essays. Remember, at the end of the day as long as the bot's ra…

Your last paragraph, and particularly the last sentence, epitomizes what is wrong with your whole thesis: the ultimate goal of the testing (and education itself, for that matter) is not to find people who can "do well on essays"; it is to develop analytical thinking.

Clearly, that referred to 'doing well on essays' correlating to good analytical thinkers. Which is does.
Post reply on HN