Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Flawed Algorithms Are Grading Millions of Students’ Essays

vice.com

Post reply on HN
Print view View original

201–210 of 302 posts

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#201

Post by choeger »

Such a stupid application of technology. It looks as if learning is completely out of fashion nowadays.

First of all, complaining about minorities getting lower grades because their English is not as sophisticated as that of others is the inversion of the idea of teaching. That feedback is actually great. We have machines that can give that feedback (e.g., grammarly)? Then use it to make everyone's writing better. Grades are just a measure of the success of learning, after all. I never got why one would not allow a student to repeat a particular test as often as they like, tbh.

Second, grading essays this way is a clear violation of the idea of teaching. What do you want the students to learn? Structure? Knowledge transfer? Grammar? Writing an essay is such a complex task it is a really too broad goal. And then naturally grading becomes quite difficult.

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#202

Post by IanSanders »

jakear wrote: post #3

“In most machine scoring states, any of the randomly selected essays with wide discrepancies between human and machine scores are referred to another human for review”. And “between 5 to 20 percent” of essays are randomly selected for human review. So the takeaway is that if you’re one of the 80-95% of (typically black or female) people who the machine scored dramatically lower, but are not selected for human review,…

>Anyone involved in the creation or adoption of these systems should be ashamed

That's the problem - there is seemingly no shame these days. People involved "saved time and money", got paid and that's it. "If I didn't do it someone else would" and all of that.

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#203

Post by sidpatil »

maze-le wrote: post #189
dagw wrote: post #183

Earlier quoted context omitted.

My guess is that soon or later people will learn how to game that AI. Already been done. http://lesperelman.com/writing-assessment-robo-grading/babel... Here's a sample essay that is complete nonsense and got a perfect score on the GRE. http://lesperelman.com/wp-content/uploads/2015/12/6-6_ScoreI...

That reads like it was written by a GPT-2 bot.

A Markov chain would have been able to produce a text like that.

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#204

Post by C1sc0cat »

maze-le wrote: post #188
piokoch wrote: post #181

Earlier quoted context omitted.

Frankly, I can't believe what I am reading. The idea that some "AI" grades essays automatically is idiotic and has nothing to do with education. Where is the place for discussions? Where is the place for ideas confrontation? Where is the place for writing style development? How this AI is supposed to grade things like repetitions (that can be either good rhetorical tool or a mistake, depending on context), etc? Who t…

>> Who the hell came out with such an idea. I'd guess this is a product of dwindling state finances and contempt for any form of real education. AI's are orders of magnitude cheaper than real teachers. They also don't form unions and wouldn't voice any opposition against changes in the curiculum. They are also pretty useless, as you have pointed out. The consequences of this policy will be postponed until the student…

Until they end up at Uni or Work and find that they haven't developed the right skills.

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#205

Post by ptx »

dcolkitt wrote: post #132
drngdds wrote: post #9

This is my first time learning that AI-graded essays are a thing. Am I the only one who thinks that's insane? I feel like you'd probably have to have an AGI to meaningfully evaluate an essay.

> I feel like you'd probably have to have an AGI to meaningfully evaluate an essay. So the reason this isn't the case, is because there are very simple metrics that tend to highly correlate with essay quality. It doesn't mean the grading-bot is actually evaluating essay quality. It's just looking for properties that are statistically associated with good essays. Remember, at the end of the day as long as the bot's ra…

I think you meant Goodhart's law: "When a measure becomes a target, it ceases to be a good measure."

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#206

Post by C1sc0cat »

tiborsaas wrote: post #159
peteradio wrote: post #49

Earlier quoted context omitted.

Based on the title alone, how would you feel if you were given bad marks due to flawed black box?

Like how I felt when I was given low grades for my ugly handwriting. It was stupid to grade it, but it guaranteed that I will never get a top score on any literature class.

Yeh I (as a dyslexic) got the same at school having my handwriting mocked by the teacher.

It took me much longer to pass my English language O level (exam taken at 16).

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#207

Post by jhanschoo »

ALittleLight wrote: post #157
dahart wrote: post #133

> Utah has been using AI as the primary scorer on its standardized tests for several years. “It was a major cost to our state to hand score, in addition to very time consuming,” said Cydnee Carter, the state’s assessment development coordinator. The automated process also allowed the state to give immediate feedback to students and teachers, she said. Yes, education takes time and costs money. Yes, not educating is b…

When I was a child I was obsessed with the "grade level" function in Microsoft Word. It was a preference you could enable on spell check to tell you the "grade level" of your writing. Every essay I wrote, I'd always force myself to reach the max "12.0" grade level. While writing I'd struggle over word choice, sentence structure, rearranging paragraphs, working on my tone etc, all in pursuit of the 12th grade way to p…

It's interesting that the tool (and system) is designed to aid people trying for the opposite result, i.e. for publicists and other authors striving to word their message to be as widely understood as possible.

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#208

Post by _0ffh »

dahart wrote: post #133

> Utah has been using AI as the primary scorer on its standardized tests for several years. “It was a major cost to our state to hand score, in addition to very time consuming,” said Cydnee Carter, the state’s assessment development coordinator. The automated process also allowed the state to give immediate feedback to students and teachers, she said. Yes, education takes time and costs money. Yes, not educating is b…

>Turns out the instructions said the essay would be scored on verbal efficiency; getting the point across clearly with the fewest words. I started playing around and realized that the more words I added, the higher the score

Apart from the fact that your story is straight up frightening, isn't this part completely backwards, too? I mean, clearly using more words to convey the same message is /less/ efficient, not more so?

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#210

Post by YeGoblynQueenne »

dahart wrote: post #133

> Utah has been using AI as the primary scorer on its standardized tests for several years. “It was a major cost to our state to hand score, in addition to very time consuming,” said Cydnee Carter, the state’s assessment development coordinator. The automated process also allowed the state to give immediate feedback to students and teachers, she said. Yes, education takes time and costs money. Yes, not educating is b…

Ho ho. Wait. You mean you were able to submit multiple versions of the essay? So that anyone can basically game the test, by submitting multiple essays until they get the best score they can wring out of it?

That is just mad.

Post reply on HN