as a precursor to grading/giving feedback on an essay, i usually pull out a Naive Bayes network for topic classification and see if their report is correctly classified. It's kind of fun and a good indicator for what i'm about to read.
New Test for Computers: Grading Essays at College Level
11–20 of 31 posts
Re: New Test for Computers: Grading Essays at College Level
#12One big advantage of this, if it ever becomes decent: iteration.
Make it available to students. They can write an essay, submit, and get feedback on it. Repeat.
It'd be difficult to get a machine to bring someone to great writing. But clear, concise prose that gets to the point? That seems reachable. And that would be an improvement over the status quo.
In other words, machines won't anytime soon be able to know that a Wodehouse is superior to an Orwell. But an Orwell to, say, a Thomas Friedman or typical ninth grader? Totally.
Re: New Test for Computers: Grading Essays at College Level
#13One big advantage of this, if it ever becomes decent: iteration. Make it available to students. They can write an essay, submit, and get feedback on it. Repeat. It'd be difficult to get a machine to bring someone to great writing. But clear, concise prose that gets to the point? That seems reachable. And that would be an improvement over the status quo. In other words, machines won't anytime soon be able to know that…
It would be fun to hook this up to, say, a Markov chain with weights adjusted by a genetic algorithm, or other sort of machine learning with text generation, to automatically produce an essay that scores high.
Re: New Test for Computers: Grading Essays at College Level
#14A quote from the article: “My first and greatest objection to the research is that they did not have any valid statistical test comparing the software directly to human graders,” said Mr. Perelman. That is a good enough reason not to use the software. If it does eventually pass such a test, I doubt that it would ever be better than the best human grader but rather it would be better than the average grader.
I'm not sure a "good" result for such software would be much progress though -- any success would be the result of students writing for a human grader. If you actually employ the software grader, students write with that in mind and start looking for "tricks" and "holes" in the algorithm that result in a good grade but diverge from what we humans regard as good writing.
Your assuming 100% automation, having a person skim for obvious bad faith essays is probably enough to combat such cheating while still saving 90% of the effort of fulling grading each essay.
Re: New Test for Computers: Grading Essays at College Level
#15So, for this to be even a feasible solution, EdX needs to show more proof of concept...like, here's a sample question, here's the 100 sample answers that were used to machine-learn against, and here's how the auto-grader graded these 10 different answers (both good and bad). Why should anyone have faith that EdX has cracked the perfect mix of machine learning and NLP and other associated technologies needed to provid…
Even if their perfect its not really good enough outside of say standardized testing because they can't integrate what they learn about the individual students from grading their papers and then use it tutor/advise/grade the student.
A holistic understanding of the individual student and how all students are doing in a course is important. It allows the professor to tailor the course or see where weaknesses in understanding are.
Also I would imagine that there are some exceptional essays which because of their unique nature would slip by the grading system. Subjectivity is a plus and a minus of having human markers.
Re: New Test for Computers: Grading Essays at College Level
#16I actually had to use something like this for a class in college… maybe 2 years ago? It was the worst thing. It would auto-grade, tell you your score, and how you could improve it. However, there was almost no way to actually respond to the grade. It would often tell you you didn't cover a specific topic, even when you had. There was no way of pointing out where you had discussed it, no way in the program of asking for a human review, anything.
All this, of course, was exacerbated by the class being run by possibly the laziest professor I had all throughout college (all essays were graded by this, all tests in class were taken by clicker, all notes were Powerpoint slides provided by the book's publisher). Possibly, in the hands of someone who actually cared about teaching, a tool like this wouldn't have been so bad—but it seems to me that someone who cared about teaching would just grade manually to begin with.
Re: New Test for Computers: Grading Essays at College Level
#17So, for this to be even a feasible solution, EdX needs to show more proof of concept...like, here's a sample question, here's the 100 sample answers that were used to machine-learn against, and here's how the auto-grader graded these 10 different answers (both good and bad). Why should anyone have faith that EdX has cracked the perfect mix of machine learning and NLP and other associated technologies needed to provid…
> Think of how much time you spent arguing with a profesor that your score deserved an 8 instead of a 6, enough points to bring you from a B to a B+ for the semester.
I spent zero time doing this, and I never even realized how widespread it was to try to negotiate grades until I helped TA a few classes. Now I see how much professors hate the students who do.
Seriously, what job is going to care that you got a B vs B+ in a class? If that subject is really so important to their mission, they are probably looking for A+ candidates anyway.
Re: New Test for Computers: Grading Essays at College Level
#18There are a few reasons I'm not excited about this sort of thing, but there's one big reason where I think it would be a distinct improvement over the status quo: standardized testing. Most standardized testing in the US, especially the ones deployed at a national scale, are designed for grading first, and testing second. This is a simple concession to feasibility: if you're trying to evaluate a few dozen million stu…
I have to disagree with you. This will make standardized testing worse, not better. With human grading, the humanities managed to avoid (to an extent) the extreme standardization of what should and should not be taught. There was still room for creativity. Now the type of writing that cannot be graded by an algorithm will get even less room in our curricula. Goodbye to poetry, satire, and other creative endeavors.
Re: New Test for Computers: Grading Essays at College Level
#19One problem: Essay quality, and that of writing in general, is subjective.
It's possible to assess objective properties of writing, but it cuts off the top end of the spectrum.
Three quarks for Muster Mark!
Sure he has not got much of a bark
And sure any he has it's all beside the mark.
Re: New Test for Computers: Grading Essays at College Level
#20So, for this to be even a feasible solution, EdX needs to show more proof of concept...like, here's a sample question, here's the 100 sample answers that were used to machine-learn against, and here's how the auto-grader graded these 10 different answers (both good and bad). Why should anyone have faith that EdX has cracked the perfect mix of machine learning and NLP and other associated technologies needed to provid…
> Think of how much time you spent arguing with a profesor that your score deserved an 8 instead of a 6, enough points to bring you from a B to a B+ for the semester. I spent zero time doing this, and I never even realized how widespread it was to try to negotiate grades until I helped TA a few classes. Now I see how much professors hate the students who do. Seriously, what job is going to care that you got a B vs B+…
GPA is, unfortunately, one metric that is used to quickly sift applicants. So one B+ over a B may not matter for one course, but can be quite significant over the course of four years.