Grading... algorithms... for essays? How/why is that even a thing? That's absolutely insane. You can't grade someone's writing skills using algorithms. That is totally counter to providing a proper education. My mind is officially boggled.
Flawed Algorithms Are Grading Millions of Students’ Essays
171–180 of 302 posts
Re: Flawed Algorithms Are Grading Millions of Students’ Essays
#172Earlier quoted context omitted.
> I feel like you'd probably have to have an AGI to meaningfully evaluate an essay. So the reason this isn't the case, is because there are very simple metrics that tend to highly correlate with essay quality. It doesn't mean the grading-bot is actually evaluating essay quality. It's just looking for properties that are statistically associated with good essays. Remember, at the end of the day as long as the bot's ra…
Strongly (but respectfully) disagree with a lot of this! This is related to current fairness-in-AI discussions. In many cases the basic problem is ML systems leverage correlations for making causal decisions. Here, there is a huge ethical difference between scoring a person based on "is this a good essay" and "do the features of this essay correlate with features of good essays". Just like there is a huge fairness an…
Which means that machine learning models need to not only be good predictors, but also be causal models to some extent.
Re: Flawed Algorithms Are Grading Millions of Students’ Essays
#173This is my first time learning that AI-graded essays are a thing. Am I the only one who thinks that's insane? I feel like you'd probably have to have an AGI to meaningfully evaluate an essay.
I totally agree that "AI" grading is totally bullshit. But, I also have plenty of experience teaching/TAing large courses, and after reading too many essays they all become semanticically saturated meaninglessness. One can not help but skim them, and grade according to a few quick heuristics. At that point one tries to be self-consistent and defensible in one's grading, but careful consideration is right out. I suspe…
The solution isn't to do a better job at grading 'meaninglessness' but to stop requiring the production of it in the first place.
One major problem with algorithmic approaches, whether automated or not, is that they become the definition of good in the context and therefore become something that cannot be argued against. And of course it makes 'teaching to the test' an even more likely outcome.
If I were a conspiracy theorist I'd attribute this to wanting a dumbed down population. Unfortunately I think it is probably the other way round, the population is already dumbed down and a belief in AI unicorns is the result.
As Aristotle said to Alexander: 'There is no royal road to geometry', and so it is with education; it's hard work for both the student and the educator and no amount of AI/ML/algorithmic snake oil will change that without also changing the meaning of the word education.
Re: Flawed Algorithms Are Grading Millions of Students’ Essays
#174Earlier quoted context omitted.
When I hear a result like "software which understands basic grammar structures can predict what grade a human would give an essay" I think my views are roughly: * 5% - cool, we could make a company that grades essays * 15% - cool, we could make a company that grades essays and sell our source code to the test-prep industry * 80% - fascinating, it sounds like the exam designers need to reevaluate what they are trying…
Whatever we decide to measure, it needs to scale to millions of essay responses each year in a way such that scores are consistent across entire states or countries. With that in mind I'd imagine it's difficult to do much more than grade on grammar and basic semantics.
No it doesn't.
And if you succeed you will simply be measuring an uninteresting but manageable subset of the problem which will then become in some people's eyes the definition of the problem.
Education is supposed to be about teaching people to think, to give them the tools with which to do it, to be able to evaluate, criticise, invent, etc.
Re: Flawed Algorithms Are Grading Millions of Students’ Essays
#175Earlier quoted context omitted.
Wow this is pretty shocking. I can understand using automated systems for something like math problems, it makes sense. There’s (usually) one right answer. But essays? This should be banned.
Wait 'til you see a kid in tears because the math answer they submitted was supposed to equal zero, but the algorithms behind the scenes are so bad that the float math failed the equality check. Note: This is not hyperbole, I have seen this exact scenario more than once. There may be a place for a well-designed one, but if it exists, I've never seen it.
I know this is just my experience but I can confirm the automated math scoring system I was using in a large US university in 2012 had bugs where many times I would enter a complex solution with fractions and it would tell me I was wrong and the correct response was some other form of the same equivalent fraction... Talk about frustrating after pouring over a question for 10 minutes.
Re: Flawed Algorithms Are Grading Millions of Students’ Essays
#176> Utah has been using AI as the primary scorer on its standardized tests for several years. “It was a major cost to our state to hand score, in addition to very time consuming,” said Cydnee Carter, the state’s assessment development coordinator. The automated process also allowed the state to give immediate feedback to students and teachers, she said. Yes, education takes time and costs money. Yes, not educating is b…
When I was a child I was obsessed with the "grade level" function in Microsoft Word. It was a preference you could enable on spell check to tell you the "grade level" of your writing. Every essay I wrote, I'd always force myself to reach the max "12.0" grade level. While writing I'd struggle over word choice, sentence structure, rearranging paragraphs, working on my tone etc, all in pursuit of the 12th grade way to p…
All that grade-level gaming paid off, I reckon! This was a funny, informative personal account of it :)
Re: Flawed Algorithms Are Grading Millions of Students’ Essays
#177Personal anecdote;
I remember taking a standardized test, can't remember if it was SAT or CSAT (Colorado pre-SAT test). This was at a time when I'm confident that humans were the graders.
I started with an intro that would be appropriate for a standard 5 paragraph essay; i.e. the thing you write when you don't know what you're talking about and you're just following a format.
In the third paragraph I took a leaf from family guy, and just interjected "WAFFLES, NICE CRISPY WAFFLES, WITH LOTS OF SYRUP." for the next page and a half, I berated the very foundation of the essay prompt, insulting it the way only an angst ridden early teen can.
... I got a 98% on the essay.
Fast forward several years. I write an essay for for an introductory college course final. My paper is returned to me with a coffee stain and a "94% - good work!" note scribbled on the top. That note was scribbled by a TA that would turn out to be my girlfriend for 2 years. One night in bed, she tilts her laptop to me, showing an article that I used as the central theme to the above essay; "can you believe this?"
"Are you joking? Of course I can believe this, it was the subject of the essay you gave me an A on 2 years ago"
She admits she didn't read past the first paragraph of anything she grades, and just bases grades on intuition based on how articulate the essays are at the outset.
...
The point I'm making:
Does AI suck at judging the amount of informative content in a student essay? YES
Do humans suck at judging the amount of informative content in a student essay? ALSO YES
Re: Flawed Algorithms Are Grading Millions of Students’ Essays
#178I can't believe that anyone would try to automatically grade essays. This is either deeply cynical or astonishingly dumb.
Re: Flawed Algorithms Are Grading Millions of Students’ Essays
#179Earlier quoted context omitted.
Hey uh, that actually seems valuable. I'd believe that ML could spot abuse that humans miss pretty well from signals like non-overt references in homework and school records, if one could come up with an adequate training set. Much more likely than teaching ML to score reasoned and creative activity in any reasonable way.
What do you think the false positive rate is likely to be?
Who cares? The positives must be evaluated by a human anyway.
Re: Flawed Algorithms Are Grading Millions of Students’ Essays
#180To me this brings up the absurdity of having essays on standardized tests. What about an essay is standardized? It's a totally nonsensical premise. This always gets made into some kind of techluminati conspiracy for the machines to ingrain structural racism whereas it's pretty clear all the algorithms fail to do is improve an already bad situation stemming from a flawed premise.
A number of states found out their schools were graduating students who genuinely could not read or write effectively. If you want to quantify that, you're forced to test it somehow. How would you test writing ability without asking them to write something?