Why Ratings Systems Don't Work • Hacker News Board

Re: Why Ratings Systems Don't Work

#61

Post by SLuijk » Wed, Aug 22, 2012, 1:41 PM UTC

guard-of-terra wrote: post #18

sp332 wrote: post #14

Earlier quoted context omitted.

It's different from having 2 histograms because it lets you distinguish people who thought it was a good movie but not re-watchable, from people who thought it was a good movie and fairly re-watchable, from people who thought it was an OK movie and not re-watchable. With a bunch of histograms you can't correlate the variables.

Why would I care about rewatchability? The torrents are full with thousands of lives on video I'll never watch, what's the point of rewatching anything but the dearest, youth-defining films?

You are not trying to gauge if you would want to re-watch the film. If other would re-watch the film, maybe, just maybe, you would enjoy watching it the first time.

Re: Why Ratings Systems Don't Work

#62

Post by thornofmight » Wed, Aug 22, 2012, 1:41 PM UTC

tokenadult wrote: post #47

The old Latin proverb "Quis custodiet ipsos custodes?" http://en.wikipedia.org/wiki/Quis_custodiet_ipsos_custodes%3... might in this context be paraphrased to "Who is rating the raters?" The hope in any online rating system is that enough people will come forward to rate something that you care about so that the people who have crazy opinions will be mere outliers among the majority of raters who share your well info…

In the context of a website like Netflix, where your recommendations and ratings for movies are based off of your history of ratings, aren't you the one rating the raters?

However, the type of rating mentioned in the OP and the type of rating on Netflix only seem to work in specific niches. I can't imagine how a website like Amazon would implement anything even close to what Goodfilms is doing.

Re: Why Ratings Systems Don't Work

#63

Post by pygy_ » Wed, Aug 22, 2012, 1:45 PM UTC

Xcelerate wrote: post #43

pygy_ wrote: post #12

Histograms are useful int the case of items with a strong love/hate split. The canonical example is the SICP ratings on Amazon: 3.5 average; 177 ratings, 96 five stars, 53 one stars. http://www.amazon.com/Structure-Interpretation-Computer-Prog...

Had to go look up that book. It's actually available free from MIT: http://mitpress.mit.edu/sicp/full-text/book/book.html

The videos of the lectures but the book authors (Gerald Sussman and Hal Abelson) are also available for free.

Pick the MPEG1 versions. They are much heavier than the MPEG4 versions, but the text on the projected computer screen is at readable. IIRC, the MPEG4 are re-encoded versions of the MPEG1, which themselves were ripped from VHS.

http://archive.org/details/mit_ocw_sicp

Re: Why Ratings Systems Don't Work

#64

Post by baby » Wed, Aug 22, 2012, 1:46 PM UTC

I thought they would come up with a better system than stars. Like a "like/dislike" or "would watch it again". But no. They still use the same system.

They just made a nice blog post to buzz around.

But I went past through that and played a bit with their app, until I found out that everything you did was posted on your timeline by default.

I erased the application from my facebook.

Re: Why Ratings Systems Don't Work

#65

Post by lttlrck » Wed, Aug 22, 2012, 1:46 PM UTC

In my experience the best films glue you to the seat. I'm fairly sure that's a universal phenomenon ;)

They are watched without pausing, skipping, rewinding, from beginning to end.

If it's not true in your case then you're not a reliable rating source.

A rating based on this can be fully automated.

No need to depend entirely on this one rating as it could be used for weighting a users subjective rating. E.g. if the film was watch over 3 evenings, then your 5 star rating is worth 1 star.

Re: Why Ratings Systems Don't Work

#66

Post by dbecker » Wed, Aug 22, 2012, 1:48 PM UTC

abrookewood wrote: post #16

vhf wrote: post #8

Earlier quoted context omitted.

This would be no different than having two histograms side by side.. Yes it would, and the article shows why and how. Scatter plots are easy to read (for comp./math. educated people). Two histograms side by side are easy to read (and find correlations) for nobody.

I'm with Henry ... the scatterplot is fundamentally no different to two histograms side-by-side (though it is marginally easier to read).

Though a scatterplot conveys less information (e.g. the correlations between the two axes), I think it takes longer to process. It also takes more screen real estate than a pair of histograms.

Re: Why Ratings Systems Don't Work

#67

Post by kittxkat » Wed, Aug 22, 2012, 1:48 PM UTC

runn1ng wrote: post #59

I still don't understand how is "rewatchability" different from "quality". If the movie is good, I would like to watch it again. What is the difference?

Fair point, I asked that myself. See ErrantX's answer[1] which IMO totally makes sense:

> Why would I care about rewatchability? Speaking as a film buff; this is actually quite a good guide to the sort of movie it is (when combined with quality). If lots of people mark it good quality, but wouldn't watch it again, that implies that you have to be in the right mood for it.

If people mark rewatchability high, even if the quality rating varies, you know it is much more easy going film.

And so on. Combining data points is good :)

[1]: http://news.ycombinator.com/item?id=4417078

Re: Why Ratings Systems Don't Work

#68

Post by sixbrx » Wed, Aug 22, 2012, 1:50 PM UTC

geelen wrote: post #3

s_henry_paulson wrote: post #2

Terrible article. Calling histograms awful, based on nothing more than an opinion. Then trying to conclude that some convoluted scatter plot system makes more sense is laughable. Not to mention, this system is still just a star rating system. This would be no different than having two histograms side by side.. assuming, of course, that you'd even want to rate different aspects of the same thing. I can't even imagine…

My point was you can't create a scatter plot from a single dimension of data, and that histograms are awful because they don't allow you to see patterns. But of course you're right, it is just my opinion.

But histograms do allow you to easily see one of the most important patterns to me, at least for technical books, which is that the book is excellent but too "hard" for some who aren't willing to put in the effort. Those books tend to have a lot of 5 star and 1 star ratings, with few in between. Check out the SICP reviews on Amazon for example. When augmented with a set of comments, it's usually easy to predict which of the two groups I'd be in.

Re: Why Ratings Systems Don't Work

#69

Post by baby » Wed, Aug 22, 2012, 1:50 PM UTC

tokenadult wrote: post #47

The old Latin proverb "Quis custodiet ipsos custodes?" http://en.wikipedia.org/wiki/Quis_custodiet_ipsos_custodes%3... might in this context be paraphrased to "Who is rating the raters?" The hope in any online rating system is that enough people will come forward to rate something that you care about so that the people who have crazy opinions will be mere outliers among the majority of raters who share your well info…

IMO the perfect solution would be to rate only by saying "I like" or "I dislike". And then writing a review (to show how much you liked/disliked the product/movie etc...)

When you first get on the website, you're asked to rate a number of products. The more you rate, the more accurate the solution becomes.

When it can couple your test with users having the same test, you know see these users ratings in priority for new products. You can even ask them why they disliked/liked something if they didn't write a review. Because their opinion matters to you now.

Re: Why Ratings Systems Don't Work

#70

Post by maxerickson » Wed, Aug 22, 2012, 1:53 PM UTC

abrookewood wrote: post #16

vhf wrote: post #8

Earlier quoted context omitted.

This would be no different than having two histograms side by side.. Yes it would, and the article shows why and how. Scatter plots are easy to read (for comp./math. educated people). Two histograms side by side are easy to read (and find correlations) for nobody.

I'm with Henry ... the scatterplot is fundamentally no different to two histograms side-by-side (though it is marginally easier to read).

A scatter plot definitely shows more information than a pair of histograms. Who knows if that additional information will be useful.

It would be nice if the article actually compared different visualizations of the same data, rather than showing histograms of 2 separate data sets and scatter plots of a 3rd data set.