Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Flawed Algorithms Are Grading Millions of Students’ Essays

vice.com

Post reply on HN
Print view View original

241–250 of 302 posts

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#241

Post by JoeAltmaier »

ptx wrote: post #205
dcolkitt wrote: post #132

Earlier quoted context omitted.

> I feel like you'd probably have to have an AGI to meaningfully evaluate an essay. So the reason this isn't the case, is because there are very simple metrics that tend to highly correlate with essay quality. It doesn't mean the grading-bot is actually evaluating essay quality. It's just looking for properties that are statistically associated with good essays. Remember, at the end of the day as long as the bot's ra…

I think you meant Goodhart's law: "When a measure becomes a target, it ceases to be a good measure."

Is that what it's called? Its referred in neural net training as overtraining.

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#242

Post by amiga_500 »

piokoch wrote: post #181
dahart wrote: post #133

> Utah has been using AI as the primary scorer on its standardized tests for several years. “It was a major cost to our state to hand score, in addition to very time consuming,” said Cydnee Carter, the state’s assessment development coordinator. The automated process also allowed the state to give immediate feedback to students and teachers, she said. Yes, education takes time and costs money. Yes, not educating is b…

Frankly, I can't believe what I am reading. The idea that some "AI" grades essays automatically is idiotic and has nothing to do with education. Where is the place for discussions? Where is the place for ideas confrontation? Where is the place for writing style development? How this AI is supposed to grade things like repetitions (that can be either good rhetorical tool or a mistake, depending on context), etc? Who t…

Is the USA tenable going forward? Your cost of everything, value of nothing culture appears to be very destructive.

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#243

Post by pintxo »

nitwit005 wrote: post #180
anm89 wrote: post #35

To me this brings up the absurdity of having essays on standardized tests. What about an essay is standardized? It's a totally nonsensical premise. This always gets made into some kind of techluminati conspiracy for the machines to ingrain structural racism whereas it's pretty clear all the algorithms fail to do is improve an already bad situation stemming from a flawed premise.

A number of states found out their schools were graduating students who genuinely could not read or write effectively. If you want to quantify that, you're forced to test it somehow. How would you test writing ability without asking them to write something?

Maybe it would be easier to test the teachers than the students then?

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#244

Post by mannykannot »

JoeAltmaier wrote: post #240
mannykannot wrote: post #238

Earlier quoted context omitted.

Your last paragraph, and particularly the last sentence, epitomizes what is wrong with your whole thesis: the ultimate goal of the testing (and education itself, for that matter) is not to find people who can "do well on essays"; it is to develop analytical thinking.

Clearly, that referred to 'doing well on essays' correlating to good analytical thinkers. Which is does.

That assumption lacks justification when the scoring does not actually measure analytical thinking. Any statistical evidence for it is suspect as a predictor of future outcomes when a high score can more easily be gamed than 'honestly' achieved.

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#245

Post by derg »

amiga_500 wrote: post #242
piokoch wrote: post #181

Earlier quoted context omitted.

Frankly, I can't believe what I am reading. The idea that some "AI" grades essays automatically is idiotic and has nothing to do with education. Where is the place for discussions? Where is the place for ideas confrontation? Where is the place for writing style development? How this AI is supposed to grade things like repetitions (that can be either good rhetorical tool or a mistake, depending on context), etc? Who t…

Is the USA tenable going forward? Your cost of everything, value of nothing culture appears to be very destructive.

Honestly, no. It's not.

Granted I already have a very fatalistic view on my future already so take my opinion with a grain of salt.

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#246

Post by [deleted] »

[deleted] wrote: post #235
jkdufair wrote: post #122

Earlier quoted context omitted.

I have a grant where were are doing just that. Implementing more or less SOTA research using fairly vanilla LSTM networks from 2-3 years ago (primarily Taghipour & Ng) to provide low stakes feedback to students on their essays in one of our teaching tools at Purdue. It’s based on research using the Kaggle ASAP database and we have found it to be pretty accurate across a variety of domains in early testing. Though som…

[deleted]

[deleted]

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#247

Post by ativzzz »

amiga_500 wrote: post #242
piokoch wrote: post #181

Earlier quoted context omitted.

Frankly, I can't believe what I am reading. The idea that some "AI" grades essays automatically is idiotic and has nothing to do with education. Where is the place for discussions? Where is the place for ideas confrontation? Where is the place for writing style development? How this AI is supposed to grade things like repetitions (that can be either good rhetorical tool or a mistake, depending on context), etc? Who t…

Is the USA tenable going forward? Your cost of everything, value of nothing culture appears to be very destructive.

It's progress built on top of an assumption of never-ending, unsustainable growth. Unfortunately for everyone else much of the world has been dragged into the rat race along with us.

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#248

Post by xamuel »

bryanrasmussen wrote: post #231
dagw wrote: post #227

Earlier quoted context omitted.

Given that the stack ranking at your future job will also be done by an "AI" (probably developed by the same company that graded you tests) this is a very useful skill to have.

Good point, I better teach my kid this skill so she can go on to a job programming one of these "AIs" idiocracy will happen not because people get any stupider, but because the bots reward the stupid ones first.

Maybe Idiocracy was right about the whole "It's got electrolytes!" meme, except swap out electrolytes for neural networks.

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#249

Post by MertsA »

yoz-y wrote: post #195
dahart wrote: post #133

> Utah has been using AI as the primary scorer on its standardized tests for several years. “It was a major cost to our state to hand score, in addition to very time consuming,” said Cydnee Carter, the state’s assessment development coordinator. The automated process also allowed the state to give immediate feedback to students and teachers, she said. Yes, education takes time and costs money. Yes, not educating is b…

> Turns out the instructions said the essay would be scored on verbal efficiency; getting the point across clearly with the fewest words. I started playing around and realized that the more words I added, the higher the score, whether they were relevant or grammatical or not. Frankly, this does not change anything from my experience in school decades ago. The teachers always said that the length does not matter and w…

Historically on the written portion of the SAT length is substantially correlated to the final score.

Re: Flawed Algorithms Are Grading Millions of Students’ Essays

#250

Post by black6 »

dvdbloc wrote: post #175
Baeocystin wrote: post #158

Earlier quoted context omitted.

Wait 'til you see a kid in tears because the math answer they submitted was supposed to equal zero, but the algorithms behind the scenes are so bad that the float math failed the equality check. Note: This is not hyperbole, I have seen this exact scenario more than once. There may be a place for a well-designed one, but if it exists, I've never seen it.

I know this is just my experience but I can confirm the automated math scoring system I was using in a large US university in 2012 had bugs where many times I would enter a complex solution with fractions and it would tell me I was wrong and the correct response was some other form of the same equivalent fraction... Talk about frustrating after pouring over a question for 10 minutes.

I remember to this day when I went round and round with a calculus teacher in high school who told me "sin(x) + 2" was incorrect. Then answer she wanted was "2 + sin(x)". I argued that addition was commutative, "2 + 3 = 3 + 2" and so forth. She wouldn't budge. She also said it was impossible to calculate the exact area under a curve, because you can't draw an infinite number of boxes under the curve.

Having an ignorant teacher is almost as bad as a flawed, black-box algorithm.

Post reply on HN