I knew I had seen a similar XKCD before. http://xkcd.com/937/
That is because xckd is the same 3 jokes repeated over and over.
It's easy to criticise, much harder to do.
Why Ratings Systems Don't Work
111–120 of 165 posts
Re: Why Ratings Systems Don't Work
#112Earlier quoted context omitted.
Why would I care about rewatchability? The torrents are full with thousands of lives on video I'll never watch, what's the point of rewatching anything but the dearest, youth-defining films?
Because "rewatchability" is not just useful to judge how much people like to watch a movie again -- it's also useful to judge how much people like the movie, find it deep, and connect with it (over pure mindless fun). So rewatchability roughly translates to "emotional connection + quality".
Now compare your point of view with the one expressed by mhellmic[1] in this very discussion. 'Rewatchability' sounded to me as something even more vague than 'quality' while reading the article. Also, my pattern for rewatching films differs greatly from both your description for rewatchability and mhellmic's, and from my own pattern for rereading books. I don't see how it can be applied to rating systems with just one additional parameter (the quality thing).
Re: Why Ratings Systems Don't Work
#113Clustering across what you've previously "liked" ala Netflix is the solution to this problem, I think. The problem is fundamentally that single number ratings don't capture that people are non-homogenous.
Re: Why Ratings Systems Don't Work
#114Earlier quoted context omitted.
Why would I care about rewatchability? The torrents are full with thousands of lives on video I'll never watch, what's the point of rewatching anything but the dearest, youth-defining films?
Because "rewatchability" is not just useful to judge how much people like to watch a movie again -- it's also useful to judge how much people like the movie, find it deep, and connect with it (over pure mindless fun). So rewatchability roughly translates to "emotional connection + quality".
For adults, this might be true, but for kids, i'm not so sure.
If you've got kids, you know they'll just watch the same flashy animated movies over and over again, even if their opinion of the content is "meh".
What would probably happen is that family films, especially animated ones, would have skewed results.
Re: Why Ratings Systems Don't Work
#115This article posted here four months ago is much better:
http://evanmiller.org/how-not-to-sort-by-average-rating.html
Discussion:
Re: Why Ratings Systems Don't Work
#116Facebook has it right with "Like" -- either you like it or not. This eliminates these review patterns:
5 stars - OMG I LOVE EVERY PRODUCT
4 stars - Love this product, but I am withholding one star because of _____
3 stars - Everything to me is just meh.
2 stars - I hate everything but this product earned 1 star for ___ and another for ____.
1 star - UPS drop-kicked my item and it arrived late, so this product is trash!
Re: Why Ratings Systems Don't Work
#117The old Latin proverb "Quis custodiet ipsos custodes?" http://en.wikipedia.org/wiki/Quis_custodiet_ipsos_custodes%3... might in this context be paraphrased to "Who is rating the raters?" The hope in any online rating system is that enough people will come forward to rate something that you care about so that the people who have crazy opinions will be mere outliers among the majority of raters who share your well info…
> "Who is rating the raters?"
Netflix does. By cross referencing your likes and dislikes against those of your fellow Netflix members, the company is able to create a meta rating system, in which the score you see for a movie is your own. You see that score because that's how much Netflix thinks you'll like it, based on how similar people liked it.
This is the only good way of going about this method. The trick is, it's easy to do this with movies, but much more difficult with product ratings and the like. Maybe this is an opportunity for someone to build something on top of Facebook or Amazon.
Re: Why Ratings Systems Don't Work
#118Glen makes a good point about how "people are good at seeing patterns" but he still gives short shrift to the histograms. I see a big difference in the histograms. The "crescent" shape of an item's histogram, like the one for Starship Troopers, is often telling on Amazon or the iTunes App Store. That either means something about the product sucks (perhaps only in a small minority of purchases, but the risk is significant) or somebody is trying to lower the rating of the item.
The more a histogram resembles an exponential increase, the better it is. The higher the exponent, the better.
Sucky:
XXXXXX
XXXX
XX
XX
XXX
XXXXXX
XXXX
XX
XX
XX
XXXXXXXX
XXXX
XX
X
XRe: Why Ratings Systems Don't Work
#119In the pre-Internet days, the way I discovered content (movies, books, music), was to take a trip to a store (Blockbuster, local bookstore, B&N, record store). There were two broad categories of content: the mainstream stuff with the primo shelf space, and the mysterious aisles of Everything Else. Judgments were based on things like cover art, in store promotional material, how many copies were still sitting on the shelf, sampling the content in-store, and recommendations from friends or store employees.
Im not sure if the success rate was any better or worse than the online star rating system these days, but it seemed more fun. However, the barrier to trying something else was also a lot higher if you made a poor choice, which might have had a side effect of narrowing one's tastes.
Re: Why Ratings Systems Don't Work
#120If something has 1000 ratings but is only 3.5 stars whereas something with 1 rating and 5 stars, what does that say about the average? Not much. It just says there's more people to rate something and have an opinion about it. Getting people to rate something is difficult in and of itself and only the people who are on the love/hate spectrum will rate something. I like the histograms as it reveals a little into the ra…
"Trustworthiness" of the reviewer is always difficult, especially with movie reviews, because there's never any accounting for taste.
I find it hard to rely on aggregated ratings for that reason.
When it came to picking movies to watch, I used to love watching Siskel and Ebert, because I knew their tastes.
If only Siskel (whose tastes were more like mine than Ebert's) gave a thumbs up, I knew there was a pretty good chance I'd at least think the movie was "ok". On the other hand, I'd be less likely to give a movie a chance if only Ebert gave a thumbs up.
These days, what I have to do is go to Rotten Tomatoes and take a sampling of four or five reviewers that I trust/like (which actually includes Roger Ebert and a few of the people he used to have as guest reviewers on Ebert & Roeper) and base my decision on that.