New Test for Computers: Grading Essays at College Level • Hacker News Board

Re: New Test for Computers: Grading Essays at College Level

#21

Post by SatvikBeri » Thu, Apr 04, 2013, 8:56 PM UTC

Of course there are a million-and-one possible problems, but the upside is also massive, especially if an automated grader is used to supplement human graders. Some examples:

-A student can iterate and improve their essays on their own using an auto-grader. They can get it up to a decent level just through iteration before having to get a human involved.

-Human graders are highly subjective

-Human graders tend to be strongly affected by factors such as "number of hours since they last ate"

-Different humans have different levels of harshness, a machine could help calibrate these

-Outside, say, the top 10% of colleges, the vast majority of human graders suck. Especially for standardized tests, they typically get paid near minimum wage and the qualifications are along the lines of "have a degree." While automated graders will probably never be as good as the best graders, they don't have to be.

So overall, I'm pretty excited. Even a half-baked solution has a lot of potential value.

Re: New Test for Computers: Grading Essays at College Level

#22

Post by notahacker » Thu, Apr 04, 2013, 9:22 PM UTC

SatvikBeri wrote: post #21

Of course there are a million-and-one possible problems, but the upside is also massive, especially if an automated grader is used to supplement human graders. Some examples: -A student can iterate and improve their essays on their own using an auto-grader. They can get it up to a decent level just through iteration before having to get a human involved. -Human graders are highly subjective -Human graders tend to be…

A student can iterate and improve their essays on their own using an auto-grader. They can get it up to a decent level just through iteration before having to get a human involved.

If awarding people degrees based on essays whose form is dictated by trial and error rather than actual comprehension and interpretation is an upside...

Re: New Test for Computers: Grading Essays at College Level

#23

Post by bitwize » Thu, Apr 04, 2013, 9:26 PM UTC

lmkg wrote: post #2

There are a few reasons I'm not excited about this sort of thing, but there's one big reason where I think it would be a distinct improvement over the status quo: standardized testing. Most standardized testing in the US, especially the ones deployed at a national scale, are designed for grading first, and testing second. This is a simple concession to feasibility: if you're trying to evaluate a few dozen million stu…

If this is deployed nationwide, how long before students start submitting carefully crafted nonsense that tricks the dopey grading algorithm into giving them an A? My bet: not long at all.

Re: New Test for Computers: Grading Essays at College Level

#24

Post by hmsimha » Thu, Apr 04, 2013, 9:27 PM UTC

danso wrote: post #20

jurassic wrote: post #17

Earlier quoted context omitted.

> Think of how much time you spent arguing with a profesor that your score deserved an 8 instead of a 6, enough points to bring you from a B to a B+ for the semester. I spent zero time doing this, and I never even realized how widespread it was to try to negotiate grades until I helped TA a few classes. Now I see how much professors hate the students who do. Seriously, what job is going to care that you got a B vs B+…

GPA is, unfortunately, one metric that is used to quickly sift applicants. So one B+ over a B may not matter for one course, but can be quite significant over the course of four years.

Additionally, certain scholarships require a minimum GPA be met. A B vs. a B- can be a huge difference to someone faced with losing their tuition check.

Re: New Test for Computers: Grading Essays at College Level

#25

Post by notahacker » Thu, Apr 04, 2013, 9:32 PM UTC

scarmig wrote: post #12

One big advantage of this, if it ever becomes decent: iteration. Make it available to students. They can write an essay, submit, and get feedback on it. Repeat. It'd be difficult to get a machine to bring someone to great writing. But clear, concise prose that gets to the point? That seems reachable. And that would be an improvement over the status quo. In other words, machines won't anytime soon be able to know that…

Machines will also "know" that the course leaders' model essays put through a bog standard article-spinner are superior to Orwell.

Re: New Test for Computers: Grading Essays at College Level

#26

Post by rayiner » Thu, Apr 04, 2013, 9:34 PM UTC

jurassic wrote: post #17

danso wrote: post #9

So, for this to be even a feasible solution, EdX needs to show more proof of concept...like, here's a sample question, here's the 100 sample answers that were used to machine-learn against, and here's how the auto-grader graded these 10 different answers (both good and bad). Why should anyone have faith that EdX has cracked the perfect mix of machine learning and NLP and other associated technologies needed to provid…

> Think of how much time you spent arguing with a profesor that your score deserved an 8 instead of a 6, enough points to bring you from a B to a B+ for the semester. I spent zero time doing this, and I never even realized how widespread it was to try to negotiate grades until I helped TA a few classes. Now I see how much professors hate the students who do. Seriously, what job is going to care that you got a B vs B+…

I've never grade-grubbed (you can't really where I went to school), but I've had to explain a B versus a B+ in an interview before.

Re: New Test for Computers: Grading Essays at College Level

#27

Post by marianne_navada » Thu, Apr 04, 2013, 9:40 PM UTC

As an educator, I appreciate technology that provides students a quicker way to get feedback, but this software only reinforces students' frustration with essays: no one reads them except the professor. Most are written to be read by one person. Artists are able to share their work, but for most majors that require writing, assignments are meant to be forgotten. This is exactly the reason why students who major in these fields graduate without a portfolio of their work.

My husband and I developed https://chalktips.com/ to solve this problem. We wanted to make essays and school work for college students engaging and shareable. Students publish booklets and slideshows as part of their assignment. Students can tweet or share their work in Facebook or Tumblr.

Last semester as part of the final, I had professionals from different fields comment on student work, and the feedback from students was amazing.

Using AI to comment on essays might be efficient and probably comparable to having ONE person grade an assignment. But it doesn't make use of the power of community. It also does not address the fundamental problem with essays: after it's graded, so what?

We currently have 1,400 users with more than 2,500 booklets and slideshows published. Students are embracing the platform. Of course, I'm a community college teacher without the clout of MIT professors. But we're hopeful that more students start demanding that they are given assignments that have utility after the course is over.

Re: New Test for Computers: Grading Essays at College Level

#28

Post by JEVLON » Thu, Apr 04, 2013, 10:15 PM UTC

Well I referenced a Harvard business school public journal and it got flagged as being from Wikipedia. So I do not believe we should let an automated system mark our essays just yet.

Re: New Test for Computers: Grading Essays at College Level

#29

Post by SatvikBeri » Thu, Apr 04, 2013, 11:25 PM UTC

notahacker wrote: post #22

SatvikBeri wrote: post #21

Of course there are a million-and-one possible problems, but the upside is also massive, especially if an automated grader is used to supplement human graders. Some examples: -A student can iterate and improve their essays on their own using an auto-grader. They can get it up to a decent level just through iteration before having to get a human involved. -Human graders are highly subjective -Human graders tend to be…

A student can iterate and improve their essays on their own using an auto-grader. They can get it up to a decent level just through iteration before having to get a human involved. If awarding people degrees based on essays whose form is dictated by trial and error rather than actual comprehension and interpretation is an upside ...

I also hold the opinion that it's much easier to learn programming if you can get feedback from the compiler in a few seconds or minutes as opposed to waiting two weeks between each compile.

Re: New Test for Computers: Grading Essays at College Level

#30

Post by solistice » Fri, Apr 05, 2013, 5:47 AM UTC

SatvikBeri wrote: post #29

notahacker wrote: post #22

Earlier quoted context omitted.

A student can iterate and improve their essays on their own using an auto-grader. They can get it up to a decent level just through iteration before having to get a human involved. If awarding people degrees based on essays whose form is dictated by trial and error rather than actual comprehension and interpretation is an upside ...

I also hold the opinion that it's much easier to learn programming if you can get feedback from the compiler in a few seconds or minutes as opposed to waiting two weeks between each compile.

You don't wait for 2 weeks to get your compiler errors? How do you handle the subjectivity of stack traces then? Dude, that's madness.

But seriously, short feedback loops facilitate learning, and if this shortens the feedback loop, then I'm exited about it.