New Test for Computers: Grading Essays at College Level
1–10 of 31 posts
Re: New Test for Computers: Grading Essays at College Level
#2There are a few reasons I'm not excited about this sort of thing, but there's one big reason where I think it would be a distinct improvement over the status quo: standardized testing.
Most standardized testing in the US, especially the ones deployed at a national scale, are designed for grading first, and testing second. This is a simple concession to feasibility: if you're trying to evaluate a few dozen million students, scantron is the only current tool that is time-efficient, cost-efficient, and consistent ("fair") at scale. The constraints imposed by that tool are significant, and tying so much incentives to those tests have warped the whole education system.
Automated essay grading, no matter how imperfect, would still be an expansion of the available testing techniques that could be deployed at the national scale. It would expand the range of skills that are measurable, and therefore incentivized to teach.
The cynical way of putting that is, no matter how shitty computers are at grading essays, they're still an improvement when the competition is multiple-choice questions. I have decided to be excited by that.
Re: New Test for Computers: Grading Essays at College Level
#3Feedback is more than a grade letter. Real teachers provide insights and direction to material like an essay to stimulate critical thought and development. How well can a computer grade system instill this type of learning on students? Can it at all?
Re: New Test for Computers: Grading Essays at College Level
#4There are a few reasons I'm not excited about this sort of thing, but there's one big reason where I think it would be a distinct improvement over the status quo: standardized testing. Most standardized testing in the US, especially the ones deployed at a national scale, are designed for grading first, and testing second. This is a simple concession to feasibility: if you're trying to evaluate a few dozen million stu…
Instead of bothering with grammar, unique personal takes on subject matter, and other complicated things, teachers can just teach students how to spew out Markov Chain incomprehensible crap peppered with high vocabulary.
Re: New Test for Computers: Grading Essays at College Level
#5A quote from the article:
“My first and greatest objection to the research is that they did not have any valid statistical test comparing the software directly to human graders,” said Mr. Perelman.
That is a good enough reason not to use the software. If it does eventually pass such a test, I doubt that it would ever be better than the best human grader but rather it would be better than the average grader.
Re: New Test for Computers: Grading Essays at College Level
#6They're facing the same problem as testing (diagnosis) in large scale health care: How to test (and teach really) without having to take on the bothersome, unscalable task of actually getting to know the student.
Wouldn't it be great if we could just figure out "education" once, code it up and then let it run on all of the students? Its easy really, all we need are identical students.
Re: New Test for Computers: Grading Essays at College Level
#7A quote from the article: “My first and greatest objection to the research is that they did not have any valid statistical test comparing the software directly to human graders,” said Mr. Perelman. That is a good enough reason not to use the software. If it does eventually pass such a test, I doubt that it would ever be better than the best human grader but rather it would be better than the average grader.
I'm not sure a "good" result for such software would be much progress though -- any success would be the result of students writing for a human grader. If you actually employ the software grader, students write with that in mind and start looking for "tricks" and "holes" in the algorithm that result in a good grade but diverge from what we humans regard as good writing.
Re: New Test for Computers: Grading Essays at College Level
#8The problematic aspects of this software are applicable to MOOCs and online education in general. Truly valuable, high-level education requires a level of precise, individually calibrated feedback in order to train students how to think and express themselves critically and rigorously (there are also obviously signalling, credentialing networking benefits that will never be replicable by MOOCs, but that's not relevant to a discussion of pure educational quality).
Computer-driven mass education will never be able to provide that level of instruction. At best, it's a tool for bringing the workforce up to the basic level of competence necessary to function in an information economy, much like how the original purpose of the public education system was to crank out semi-skilled factory workers and low-level clerks. This may be a valuable end - it may help to stem the commoditization of the BA and the attendant tsunami of student debt - but there needs to be wider acknowledgement that it is actually fulfilling a purpose that is different from that of traditional higher education.
Re: New Test for Computers: Grading Essays at College Level
#9So, for this to be even a feasible solution, EdX needs to show more proof of concept...like, here's a sample question, here's the 100 sample answers that were used to machine-learn against, and here's how the auto-grader graded these 10 different answers (both good and bad).
Why should anyone have faith that EdX has cracked the perfect mix of machine learning and NLP and other associated technologies needed to provide accurate assessments of essays? Even Google from time to time has trouble guessing intent. Wolfram Alpha even more so. If the engineers at these companies can't get it always right -- and it's not just engineering talent, but data and data analysis -- why should a school entrust one of its most important functions to EdX?
Grading is something critical to get right, not just, "almost there." Think of how much time you spent arguing with a profesor that your score deserved an 8 instead of a 6, enough points to bring you from a B to a B+ for the semester...Think of the incentive to do so (career prospects). If the machine is ever wildly off just even in one case, would you ever take its assessments as gospel? Multiply yourself by 20 or 50 or whatever a usual professor's lecture load is, and now a ton of lag has been introduced into the grading workflow.
Obviously, there are ways to mitigate this. One would be to write questions that are so narrowly focused that there are very clearly right and wrong answers...Which of course raises the problem of: why not just give everyone multiple choice tests, then?
The sad thing is is that even if these machine graders were empirically better than human graders, they can't just be better, they have to be almost perfect. If a plane's autopilot failed, on the whole, 1 out of a 1000 times compared to 5 out of a 1000 times for human pilots...how much more pissed do you think victim families are going to be when they find out their loved ones died because of a algorithmic malfunction rather than just pilot error? People, for obvious reasons, don't like thinking of their work or their destinies being defined by deterministic machines...so if those machines aren't 99.99% right, then the fallout and pushback may cost schools more than the savings in professor-grading time.
Re: New Test for Computers: Grading Essays at College Level
#10So, for this to be even a feasible solution, EdX needs to show more proof of concept...like, here's a sample question, here's the 100 sample answers that were used to machine-learn against, and here's how the auto-grader graded these 10 different answers (both good and bad). Why should anyone have faith that EdX has cracked the perfect mix of machine learning and NLP and other associated technologies needed to provid…
Critical precision in grading is only necessary in competitive high-stakes academic environments where everyone is striving for excellence (or at least a shot at belonging to the social or technical elite). If you consider an environment where the point is simply to train a crowd of semi-skilled workers to a level of basic white-collar functionality (which is what most mid- to lower-tier colleges are approaching, and what MOOCs will probably become), widespread use of this technology becomes much more plausible and acceptable.