Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Algorithm Didn’t Like My Essay

nytimes.com

Post reply on HN
Print view View original

51–59 of 59 posts

Re: The Algorithm Didn’t Like My Essay

#51

Post by _delirium »

jacktoole1 wrote: post #7
kenrikm wrote: post #2

If it's ok for them to use an algorithm to grade essays then it would be ok for students to use an algorithm to write the papers no? How is it different from using a calculator? it's a tool right? mathematics used to only be only pen and paper. (I know it says it would still be graded by people, I'm making a point) Writing is an artform and as such should be treated and graded as one. Sometimes you need to use langua…

I upvoted you for the interesting line of thought, although I disagree with the premise that acceptability of grading algorithmically implies acceptability of writing algorithmically. In CS, we often grade assignments automatically, although we expect students to write the assignments by hand. Although there are many classes where this would be a bad idea, there are some where it works quite well - if the problems ar…

One difference with CS algorithmic grading is that it's much more specific and causally connected to outcomes. It's not looking for statistical correlates of the input text and a notion of "good program", but is actually finding things like compile errors and failed testcases. And, crucially, it can also report those to the student: your program is wrong because when fed 5 it gives an output of 23 when the output should be 25.

It's probably possible to grade student programs like this essay-grading system works: instead of feeding the programs to a compiler and running against test-cases, just turn the source code into a bag-of-words representation and feed it to a statistical classifier. There are almost certainly some statistical correlates between the source code of programs and whether they're good/bad submissions. But the results would be both less reliable and less informative as feedback.

Re: The Algorithm Didn’t Like My Essay

#52

Post by ludflu »

InclinedPlane wrote: post #31
adrianN wrote: post #24

Earlier quoted context omitted.

But this article advocates using automated grading to reduce costs and free up teacher time for better things.

Cost is orthogonal to educational quality. As I said, this isn't an example of reducing costs so much as an example of the continual dehumanization of the education system. Any cost savings will just be gobbled up by the entrenched interests in the system, bureaucrats, administrators, etc. And any efficiencies making it easier to treat students like cattle and push "education" more and more toward useless rote memori…

> Cost is orthogonal to educational quality.

I've heard this (anecodotal) argument alot. Is there any empirical backing for it, and within what parameters does it hold true? My mother taught at a very expensive private school, which enabled me to switch from a lousy public school . The difference in the quality of teaching was dramatic. Private school teachers don't get paid alot more than public school teachers, and often less, although they are compensated with fringe benefits like I enjoyed, and prestige. In any case, elite private schools certainly get better results, which is why rich people pay alot of money to send their children to elite private schools. (Lots of this is attributable to the selection process, which is part of what parents are paying for.)

In contrast, public inner city schools can't choose their student population, and lack basic things like, books, paper, and air conditioning in the middle of the summer. I'm sure that proper and equitable funding of public education wouldn't solve all the problems in US public schools, but starving them of much needed resources certainly exacerbates the problems they do have.

Re: The Algorithm Didn’t Like My Essay

#53

Post by gammarator »

_delirium wrote: post #50
gwern wrote: post #48

Earlier quoted context omitted.

Goodhart's law?

Aha, almost it; I was thinking of the very similar Campbell's law, which your mention of Goodhart's law led me to. Somehow no combination of search terms got me to either of those when I was trying to come up with the name, though...

Very interesting, and a useful concept. Both grades and standarized testing themselves are subject to these pressures, obviously.

Re: The Algorithm Didn’t Like My Essay

#54

Post by [deleted] »

fiatmoney wrote: post #46

As long as we're training algorithms to recognize correlates of "high-quality writing" rather than high-quality writing itself, why not use as many predictive features as possible? I'll bet parental income and education level, average home price in the school district, and the percentage of students at the school receiving free or reduced-price lunches, are incredibly correlated with writing quality.

[deleted]

Re: The Algorithm Didn’t Like My Essay

#55

Post by ericd »

yummyfajitas wrote: post #47
ericd wrote: post #34

Earlier quoted context omitted.

I'm sorry, but this is absurd, and it makes me very angry. If I got a whiff of this attitude from you as my teacher, you would not get an ounce of my respect as a student. Why should I spend time trying to produce good work to impress you if you've already decided that what I produce will be shit? If you take the ridiculous view that my attention to grammatical detail, essay length, and variety of vocabulary used are…

...you've already decided that what I produce will be shit... What he said: " Most of these kids are straight-up idiots..." He hasn't already decided, he just came up with a prior.

He said it was true right up to the top 1%. If a teacher has disdain for the majority of his students, everything I said there holds.

Re: The Algorithm Didn’t Like My Essay

#56

Post by jacktoole1 »

_delirium wrote: post #51
jacktoole1 wrote: post #7

Earlier quoted context omitted.

I upvoted you for the interesting line of thought, although I disagree with the premise that acceptability of grading algorithmically implies acceptability of writing algorithmically. In CS, we often grade assignments automatically, although we expect students to write the assignments by hand. Although there are many classes where this would be a bad idea, there are some where it works quite well - if the problems ar…

One difference with CS algorithmic grading is that it's much more specific and causally connected to outcomes. It's not looking for statistical correlates of the input text and a notion of "good program", but is actually finding things like compile errors and failed testcases. And, crucially, it can also report those to the student: your program is wrong because when fed 5 it gives an output of 23 when the output sho…

Good point; I agree this makes code much more naturally machine graded than essays, and a statistical classifier for source code sounds disturbing (although it might do better on grading style than the usual autograding does).

Re: The Algorithm Didn’t Like My Essay

#57

Post by jacktoole1 »

ludflu wrote: post #52
InclinedPlane wrote: post #31

Earlier quoted context omitted.

Cost is orthogonal to educational quality. As I said, this isn't an example of reducing costs so much as an example of the continual dehumanization of the education system. Any cost savings will just be gobbled up by the entrenched interests in the system, bureaucrats, administrators, etc. And any efficiencies making it easier to treat students like cattle and push "education" more and more toward useless rote memori…

> Cost is orthogonal to educational quality. I've heard this (anecodotal) argument alot. Is there any empirical backing for it, and within what parameters does it hold true? My mother taught at a very expensive private school, which enabled me to switch from a lousy public school . The difference in the quality of teaching was dramatic. Private school teachers don't get paid alot more than public school teachers, and…

The countries that perform well on international standardized tests generally spend much less money per student, but a teaching job is much more prestigious than it is in the U.S.

I personally think that explanation fits both the "cost is orthogonal to education quality" argument of the parent, and your point that elite private schools have significantly better education quality, though I do not know offhand of studies about this.

Re: The Algorithm Didn’t Like My Essay

#58

Post by rwallace »

kenrikm wrote: post #2

If it's ok for them to use an algorithm to grade essays then it would be ok for students to use an algorithm to write the papers no? How is it different from using a calculator? it's a tool right? mathematics used to only be only pen and paper. (I know it says it would still be graded by people, I'm making a point) Writing is an artform and as such should be treated and graded as one. Sometimes you need to use langua…

> then it would be ok for students to use an algorithm to write the papers no?

If a student used an essay-writing algorithm he designed himself, I would certainly consider that okay.

Re: The Algorithm Didn’t Like My Essay

#59

Post by feral »

yummyfajitas wrote: post #45
feral wrote: post #41

Myself and a colleague entered this competition. We came 9th. We were doing this for fun, and aren't experts in the domain, but I think our score was within 'diminishing returns' of the better teams. There are a couple of things to realize about this challenge: I wouldn't conceptualise the challenge as trying to find features of good essays. It is more about trying to find features that are predictive of the essay be…

Is a longer essay really a better one? No. But, at the level the students were at, it just so happened that the students who were able to write better, also were able to write longer essays. It could also be the case that length is one of the features your human graders are using to grade essays. I.e., it might really be causal, rather than merely correlated. In my (anecdotal) experience, teachers certainly do this.…

Its possible. More generally, its also possible the human graders were doing a bad job; the ML system can only learn 'essay quality' to the extent that the training data reflects it.

However, the kaggle supplied 'straw-man' benchmark, which worked solely based on the count of characters and words in the essay, had an score of .647 with the training data. (The score metric used isnt trivial to interpret - it was 'Weighted Mean Quadratic Weighted Kappa' - but for reference the best entries had a score of ~.8 at the end)

The score of .647, just using length, is quite high. For length to have this powerful a causal predictive effect, the human graders would have to be weighting for length, as a feature, very heavily.

I can't rule that out; but I think its highly likely a major component of the predictive effect of length was correlative, rather than causal.

Post reply on HN