Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Algorithm Didn’t Like My Essay

nytimes.com

Post reply on HN
Print view View original

31–40 of 59 posts

Re: The Algorithm Didn’t Like My Essay

#31

Post by InclinedPlane »

adrianN wrote: post #24
InclinedPlane wrote: post #18

We keep paying more and more per student for public education. And what do we get in return? Robo-grading and prison-like treatment of students (closed campuses, zero tolerance taken to ridiculous extent). And again we come around to the notion that if only we'd spend more everything would get better, which has worked so well every time it's happened in the past. The increasing reliance on automated tools and standar…

But this article advocates using automated grading to reduce costs and free up teacher time for better things.

Cost is orthogonal to educational quality. As I said, this isn't an example of reducing costs so much as an example of the continual dehumanization of the education system. Any cost savings will just be gobbled up by the entrenched interests in the system, bureaucrats, administrators, etc. And any efficiencies making it easier to treat students like cattle and push "education" more and more toward useless rote memorization (teaching to the test) and busywork. These are not good things.

Re: The Algorithm Didn’t Like My Essay

#32

Post by willyt »

biot wrote: post #29
CapitalistCartr wrote: post #11

Earlier quoted context omitted.

It does not necessarily lead to an arms race. Once the students learn to write to the gradind software, the schools can claim the students are getting better, and point to an 'objective, unbiased' measuring stick. Everyone wins except for society, i.e. sll of us.

I'm assuming there will be competition among the companies who develop such software where, periodically, schools will re-evaluate their provider and select the best one. Company A will demonstrate their competitive advantage by showing how Company B's software incorrectly grades as "Excellent" an essay that looks like English but reads as gibberish. Company B will correct their software, but meanwhile Company C has…

You are forgetting the negative effect of the ridiculous ranking systems that determine teachers performance evaluations. Why would a school select software which will make thier grades go down? If your grades go down it decimates the schools ranking because these are based on year on year improvements. You would select the software which makes it easiest to teach to the test so the kids 'do well'

Re: The Algorithm Didn’t Like My Essay

#33

Post by Jacobi »

andrewcooke wrote: post #19
gammarator wrote: post #15

Earlier quoted context omitted.

You may be overestimating the sophistication of these algorithms. At least for this training set, my algorithm rewarded the length of the essay most of all (something like 65% of the total prediction). The only other significant factors were misspellings and prevalence of certain parts of speech. That model matched the accuracy of human graders and several commercial essay grading packages. Students reverse-engineeri…

currently, if a student writes nonsense, there's a fairly significant chance that they will be caught and penalised. a human can detect nonsense in three minutes. in contrast, i suspect algorithmic approaches can be gamed more easily because they don't adapt in the same way. they're not solving the hard ai problem; they're grading essays (currently) written for a human reviewer. for example, what happens if a child l…

Yes, they could even learn by heart the lyrics of a song and adapt it to the context.

Re: The Algorithm Didn’t Like My Essay

#34

Post by ericd »

bermanoid wrote: post #20
majormajor wrote: post #8

Earlier quoted context omitted.

For the standardized test discussed in the article, it says the human graders gave three minutes, on average, to reading each essay. These tests are not wanting art. Writing one of these essays is no more art than writing a basic linked list implementation. It's not concerned with how good a writer a kid is, but that the kid can write at all. Edit: basically, the sheer scale of the grading required for a standardized…

"Art" is not even a question - these tests typically apply to high school kids, and you're lucky if bare-bones basic communication is possible there. Most of these kids are straight-up idiots, and I say this having spent several years teaching them. I'd guess that the algorithm "map number of Microsoft Word reported spelling errors from (0 -> 800) to (20 -> 400)" does a pretty damn good job of guessing students' SAT…

I'm sorry, but this is absurd, and it makes me very angry. If I got a whiff of this attitude from you as my teacher, you would not get an ounce of my respect as a student. Why should I spend time trying to produce good work to impress you if you've already decided that what I produce will be shit? If you take the ridiculous view that my attention to grammatical detail, essay length, and variety of vocabulary used are the most important things about my writing, and enough to determine my grade?

I say this as someone who graduated at the top of my high school class - if you can't get me to work hard for you, you're pretty screwed with the average student. It's no wonder that the stuff they wrote for you was crap.

Re: The Algorithm Didn’t Like My Essay

#35

Post by SagelyGuru »

At best, these algorithms can only pick on obvious spelling, grammar, punctuation and style errors. They have no concept of semantics. Therefore the students will have to polish the form and abandon the content.

To prove this point, I suggest setting up a testing competition, this time for algorithms to generate meaningless 'essays'. I expect their easy success obtaining full marks.

So, what will be the educational outcome of this? In three simple words: more dumbing down.

Is this another example of the obsession of officials everywhere with the pedestrian form at the expense of original content? I think it tells us a lot about their own limited mentality.

Re: The Algorithm Didn’t Like My Essay

#36

Post by ericd »

This article is coming to the wrong conclusion from the facts given. That the grades given by these algorithms matched the grades given by human essay reviewers on standardized tests almost exactly doesn't mean that our algorithms are great and that they should be rolled out to normal school environments to help students learn to write. It means that our method of grading essays on standardized tests is absolute garbage, because the algorithms can't judge semantic meaning or depth of insight. We're basically grading people on how prepared they are for writing About.com content farm articles.

Re: The Algorithm Didn’t Like My Essay

#37

Post by PaulHoule »

I was told by the head search engineer for a well-known job board site that they'd trained a bag-of-words classifier to find badly written resumes that would benefit from professional editing.

They'd found it was one of the easiest text classification problems that had ever been tried. With a very small training set they'd attained off-the-charts accuracys.

Re: The Algorithm Didn’t Like My Essay

#38

Post by _delirium »

I wonder how the state-of-the-art in algorithmic essay grading compares to the state-of-the-art in scoring the quality of online content. Does Google have content-quality-estimating algorithms that could improve algorithmic essay grading?

Re: The Algorithm Didn’t Like My Essay

#39

Post by DougBTX »

dhx wrote: post #30

How well did the algorithms detect insightful analysis, deep understanding beyond the immediate subject matter, factual correctness, salience and an ability to write to a specific audience? When reading articles on the Internet I look out for superficial factors including: ‣ Misuse of U+0022 typewriter double quotation marks in place of U+201C and U+201D double quotation marks ‣ Misuse of U+002D hyphens in place of U…

How well did the algorithms detect insightful analysis, deep understanding beyond the immediate subject matter, factual correctness, salience and an ability to write to a specific audience?

I'm far from well informed, but my understanding of standardised tests is that the standard specifies the algorithm, which already ignores your good points above to achieve standardised grading. All that really changes is whether a human or a robot executes the algorithm, the human insight has already been squeezed out of the system.

Re: The Algorithm Didn’t Like My Essay

#40

Post by keithpeter »

"And humans are not necessarily ideal graders: they provide an average of only three minutes of attention per essay, Ms. Chow says. "

Well, I'm hardly surprised that an algorithm can duplicate that kind of superficial marking of an essay!

Good heavens, what are you doing to your children in the US?

Post reply on HN